Jul 30 15:09:31.541914 ip-10-0-142-244 systemd[1]: Starting Kubernetes Kubelet... Jul 30 15:09:31.941254 ip-10-0-142-244 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 15:09:31.941254 ip-10-0-142-244 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 30 15:09:31.941254 ip-10-0-142-244 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 15:09:31.941254 ip-10-0-142-244 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 30 15:09:31.941254 ip-10-0-142-244 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 15:09:31.941872 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.941455 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 30 15:09:31.945025 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945009 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 15:09:31.945025 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945024 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945028 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945033 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945036 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945039 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945042 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945045 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945048 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945051 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945053 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945057 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945059 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945062 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945064 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945067 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945070 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945075 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945078 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945080 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945083 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 15:09:31.945086 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945086 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945088 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945091 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945094 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945097 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945100 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945103 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945105 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945108 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945110 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945113 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945115 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945118 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945122 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945126 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945129 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945132 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945134 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945146 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 15:09:31.945596 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945149 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945152 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945155 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945157 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945160 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945162 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945165 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945168 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945170 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945173 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945175 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945178 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945180 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945182 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945185 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945188 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945191 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945194 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945196 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945199 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 15:09:31.946060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945201 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945204 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945207 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945209 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945212 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945214 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945217 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945219 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945222 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945224 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945227 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945229 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945231 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945234 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945236 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945239 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945242 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945244 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945247 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945250 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 15:09:31.946544 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945253 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945255 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945258 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945260 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945263 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945266 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945269 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945272 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945275 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945278 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945280 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945283 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945286 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945288 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945292 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945295 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945298 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945300 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.945303 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 15:09:31.947042 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946846 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946855 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946858 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946861 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946864 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946867 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946870 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946872 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946875 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946878 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946880 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946883 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946886 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946888 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946892 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946894 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946897 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946900 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946903 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 15:09:31.947495 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946907 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946912 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946915 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946918 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946920 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946924 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946927 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946930 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946933 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946936 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946939 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946942 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946944 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946947 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946950 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946952 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946955 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946957 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946961 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946963 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 15:09:31.948013 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946967 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946969 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946972 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946975 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946977 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946980 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946982 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946984 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946987 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946990 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946993 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946996 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.946999 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947002 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947005 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947007 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947010 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947012 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947015 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947018 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 15:09:31.948488 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947020 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947023 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947025 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947028 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947031 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947033 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947036 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947038 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947041 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947043 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947046 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947049 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947052 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947055 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947058 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947060 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947063 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947065 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947068 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947070 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 15:09:31.948995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947073 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947077 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947079 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947082 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947084 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947087 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947089 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947092 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947094 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947098 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947101 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947104 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947107 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947110 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947112 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947115 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947117 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947120 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947123 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 15:09:31.949478 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947125 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947486 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947493 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947497 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947500 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947503 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947506 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947508 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947511 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947514 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947516 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947519 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947521 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947524 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947528 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947531 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947534 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947536 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947539 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 15:09:31.949952 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947541 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947544 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947546 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947549 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947552 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947554 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947557 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947559 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947562 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947564 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947567 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947569 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947571 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947574 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947577 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947580 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947582 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947584 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947587 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947591 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 15:09:31.950407 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947594 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947597 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947601 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947604 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947607 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947610 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947613 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947616 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947619 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947622 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947625 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947628 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947630 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947633 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947636 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947652 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947655 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947657 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947660 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947662 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 15:09:31.950902 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947665 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947667 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947670 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947672 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947674 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947677 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947680 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947683 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947687 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947689 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947692 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947695 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947697 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947700 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947703 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947706 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947708 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947711 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947713 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 15:09:31.951411 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947716 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947719 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947723 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947725 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947728 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947730 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947732 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947735 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947738 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947740 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947742 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947745 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947747 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947750 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947752 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947754 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947757 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947759 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947762 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947765 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 15:09:31.951914 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947767 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.947770 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947838 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947844 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947851 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947857 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947863 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947868 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947879 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947884 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947887 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947890 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947893 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947897 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947899 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947903 2579 flags.go:64] FLAG: --cgroup-root="" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947906 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947909 2579 flags.go:64] FLAG: --client-ca-file="" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947912 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947915 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947921 2579 flags.go:64] FLAG: --cluster-domain="" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947924 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947927 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947930 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 30 15:09:31.952397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947933 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947936 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947942 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947945 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947948 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947951 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947954 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947957 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947960 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947963 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947967 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947970 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947973 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947976 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947978 2579 flags.go:64] FLAG: --enable-server="true" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947981 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947986 2579 flags.go:64] FLAG: --event-burst="100" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947989 2579 flags.go:64] FLAG: --event-qps="50" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947991 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947994 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.947997 2579 flags.go:64] FLAG: --eviction-hard="" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948001 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948004 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948006 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948010 2579 flags.go:64] FLAG: --eviction-soft="" Jul 30 15:09:31.952980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948013 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948015 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948018 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948021 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948024 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948027 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948030 2579 flags.go:64] FLAG: --feature-gates="" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948034 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948036 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948040 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948044 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948047 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948050 2579 flags.go:64] FLAG: --help="false" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948053 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-142-244.ec2.internal" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948056 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948059 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948062 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948065 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948069 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948071 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948074 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948077 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948080 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 30 15:09:31.953736 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948082 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948085 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948088 2579 flags.go:64] FLAG: --kube-reserved="" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948091 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948094 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948097 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948099 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948102 2579 flags.go:64] FLAG: --lock-file="" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948105 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948108 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948111 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948116 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948119 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948121 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948124 2579 flags.go:64] FLAG: --logging-format="text" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948127 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948130 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948133 2579 flags.go:64] FLAG: --manifest-url="" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948138 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948144 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948147 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948151 2579 flags.go:64] FLAG: --max-pods="110" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948153 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948156 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948159 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 30 15:09:31.954330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948162 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948165 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948168 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948171 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948177 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948180 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948183 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948186 2579 flags.go:64] FLAG: --pod-cidr="" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948189 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948195 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948198 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948201 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948204 2579 flags.go:64] FLAG: --port="10250" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948207 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948210 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08d2e80c5efbe5ea6" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948213 2579 flags.go:64] FLAG: --qos-reserved="" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948215 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948219 2579 flags.go:64] FLAG: --register-node="true" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948222 2579 flags.go:64] FLAG: --register-with-taints="" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948225 2579 flags.go:64] FLAG: --registry-burst="10" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948228 2579 flags.go:64] FLAG: --registry-qps="5" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948230 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948233 2579 flags.go:64] FLAG: --reserved-memory="" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948236 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948239 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 30 15:09:31.954938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948243 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948246 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948250 2579 flags.go:64] FLAG: --runonce="false" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948253 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948256 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948259 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948262 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948265 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948268 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948271 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948274 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948277 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948280 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948283 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948286 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948288 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948291 2579 flags.go:64] FLAG: --system-cgroups="" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948294 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948299 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948302 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948305 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948308 2579 flags.go:64] FLAG: --tls-min-version="" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948311 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948314 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948316 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 30 15:09:31.955612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948320 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948322 2579 flags.go:64] FLAG: --v="2" Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948326 2579 flags.go:64] FLAG: --version="false" Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948330 2579 flags.go:64] FLAG: --vmodule="" Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948334 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.948337 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948447 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948452 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948454 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948459 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948462 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948465 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948468 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948471 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948474 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948476 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948479 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948481 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948484 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948487 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 15:09:31.956234 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948489 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948494 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948497 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948501 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948504 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948507 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948511 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948514 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948517 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948520 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948522 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948525 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948527 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948531 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948533 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948536 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948539 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948541 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948544 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 15:09:31.956732 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948547 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948550 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948553 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948556 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948559 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948561 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948564 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948566 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948569 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948571 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948574 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948576 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948579 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948581 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948584 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948587 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948589 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948592 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948594 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948597 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 15:09:31.957228 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948599 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948603 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948607 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948610 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948612 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948615 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948619 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948622 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948625 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948628 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948631 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948633 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948637 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948651 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948655 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948658 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948660 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948663 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948666 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948668 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 15:09:31.957735 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948671 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948674 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948676 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948679 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948681 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948684 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948686 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948689 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948691 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948694 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948696 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948699 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948701 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948704 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948706 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948709 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948711 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948714 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948716 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 15:09:31.958216 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948718 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 15:09:31.958680 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948724 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 15:09:31.958680 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948727 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 15:09:31.958680 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948729 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 15:09:31.958680 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948732 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 15:09:31.958680 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948735 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 15:09:31.958680 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.948738 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 15:09:31.958680 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.949430 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 15:09:31.958680 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.956017 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 30 15:09:31.958680 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.956033 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 30 15:09:31.958680 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.956054 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 30 15:09:31.958680 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.956064 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 30 15:09:31.958680 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956129 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 15:09:31.958680 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956134 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 15:09:31.958680 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956137 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 15:09:31.958680 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956140 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956143 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956146 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956149 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956152 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956155 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956157 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956160 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956162 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956165 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956168 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956171 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956173 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956176 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956179 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956181 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956184 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956187 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956190 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956192 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 15:09:31.959058 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956195 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956197 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956200 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956203 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956207 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956211 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956216 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956219 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956222 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956225 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956228 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956231 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956234 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956237 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956240 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956242 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956245 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956248 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956251 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 15:09:31.959558 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956254 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956256 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956259 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956261 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956264 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956267 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956269 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956272 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956274 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956277 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956280 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956283 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956285 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956288 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956290 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956293 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956295 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956298 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956301 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956304 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 15:09:31.960060 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956307 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956309 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956312 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956314 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956317 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956320 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956322 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956325 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956327 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956330 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956332 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956335 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956337 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956340 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956342 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956345 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956348 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956351 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956355 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956358 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 15:09:31.960574 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956361 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 15:09:31.961064 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956364 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 15:09:31.961064 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956367 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 15:09:31.961064 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956370 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 15:09:31.961064 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956372 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 15:09:31.961064 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956375 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 15:09:31.961064 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956379 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 15:09:31.961064 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956383 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 15:09:31.961064 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956385 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 15:09:31.961064 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956389 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 15:09:31.961064 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956391 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 15:09:31.961064 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956395 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 15:09:31.961064 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956398 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 15:09:31.961064 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956401 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 15:09:31.961064 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956403 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 15:09:31.961064 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956406 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 15:09:31.961064 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956409 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 15:09:31.961447 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.956414 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 15:09:31.961447 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956546 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 15:09:31.961447 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956551 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 15:09:31.961447 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956555 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 15:09:31.961447 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956558 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 15:09:31.961447 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956561 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 15:09:31.961447 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956564 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 15:09:31.961447 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956567 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 15:09:31.961447 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956569 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 15:09:31.961447 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956572 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 15:09:31.961447 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956574 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 15:09:31.961447 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956577 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 15:09:31.961447 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956580 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 15:09:31.961447 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956582 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 15:09:31.961447 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956585 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956588 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956591 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956593 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956595 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956598 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956600 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956603 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956605 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956608 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956610 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956613 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956616 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956618 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956621 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956623 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956626 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956629 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956631 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956634 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 15:09:31.961839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956637 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956653 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956657 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956660 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956662 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956665 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956667 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956670 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956672 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956675 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956677 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956680 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956682 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956685 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956688 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956691 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956693 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956696 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956698 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956701 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 15:09:31.962323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956704 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956706 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956709 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956711 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956714 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956716 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956719 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956721 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956724 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956727 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956729 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956732 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956735 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956737 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956740 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956745 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956748 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956751 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956753 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956756 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 15:09:31.962891 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956759 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956762 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956764 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956767 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956769 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956772 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956774 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956777 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956779 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956783 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956787 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956790 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956792 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956795 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956797 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956800 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956802 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956805 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956807 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 15:09:31.963372 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956811 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 15:09:31.963839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956815 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 15:09:31.963839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956818 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 15:09:31.963839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956821 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 15:09:31.963839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956824 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 15:09:31.963839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956826 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 15:09:31.963839 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:31.956829 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 15:09:31.963839 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.956833 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 15:09:31.963839 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.956942 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 30 15:09:31.963839 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.959134 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 30 15:09:31.963839 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.960423 2579 server.go:1019] "Starting client certificate rotation" Jul 30 15:09:31.963839 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.960521 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 15:09:31.963839 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.960558 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 15:09:31.963839 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.962927 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 15:09:31.965667 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.965651 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 15:09:31.976178 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.976155 2579 log.go:25] "Validated CRI v1 runtime API" Jul 30 15:09:31.978271 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.978253 2579 log.go:25] "Validated CRI v1 image API" Jul 30 15:09:31.979059 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.979038 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 30 15:09:31.981317 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.981300 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 cd318bee-eca4-409b-b639-00eacb0b3fb2:/dev/nvme0n1p4 e682e507-4de4-4eab-8dfb-e94926e9bb1e:/dev/nvme0n1p3] Jul 30 15:09:31.981377 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.981317 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 30 15:09:31.986503 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.986403 2579 manager.go:216] Machine: {Timestamp:2026-07-30 15:09:31.9849796 +0000 UTC m=+0.326264368 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102777 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec254eb907387b0ec5e85a851f8b0451 SystemUUID:ec254eb9-0738-7b0e-c5e8-5a851f8b0451 BootID:8019b508-70bb-4fa3-b0c3-207d0bc72db4 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:38:28:18:8a:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:38:28:18:8a:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:e7:d6:8b:1e:1e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 30 15:09:31.986503 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.986500 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 30 15:09:31.986600 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.986569 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 30 15:09:31.989778 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.989751 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 30 15:09:31.989901 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.989780 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-244.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 30 15:09:31.990515 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.990499 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 15:09:31.990666 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.990655 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 30 15:09:31.990699 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.990668 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 30 15:09:31.990699 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.990676 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 15:09:31.991332 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.991322 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 15:09:31.992239 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.992230 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 30 15:09:31.994177 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.994166 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 30 15:09:31.994288 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.994278 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 30 15:09:31.994317 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.994309 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 30 15:09:31.994344 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.994322 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 30 15:09:31.994370 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.994346 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 30 15:09:31.994370 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.994361 2579 kubelet.go:398] "Adding apiserver pod source" Jul 30 15:09:31.994428 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.994382 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 30 15:09:31.994676 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.994665 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 15:09:31.994719 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.994682 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 15:09:31.996773 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.996756 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 30 15:09:31.998890 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.998877 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 30 15:09:31.998939 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:31.998907 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 30 15:09:32.000752 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.000740 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 30 15:09:32.000794 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.000756 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 30 15:09:32.000794 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.000762 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 30 15:09:32.000794 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.000767 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 30 15:09:32.000794 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.000775 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 30 15:09:32.000794 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.000780 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 30 15:09:32.000794 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.000788 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 30 15:09:32.000794 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.000793 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 30 15:09:32.000966 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.000800 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 30 15:09:32.000966 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.000805 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 30 15:09:32.000966 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.000820 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 30 15:09:32.000966 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.000825 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 30 15:09:32.001595 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.001586 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 30 15:09:32.001632 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.001596 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 30 15:09:32.004679 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.004666 2579 server.go:1268] "Started kubelet" Jul 30 15:09:32.004751 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.004728 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 30 15:09:32.004810 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.004747 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 30 15:09:32.004858 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.004812 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 30 15:09:32.005348 ip-10-0-142-244 systemd[1]: Started Kubernetes Kubelet. Jul 30 15:09:32.008985 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.008911 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 30 15:09:32.009314 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.009290 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-244.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 30 15:09:32.009476 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.009450 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-244.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 30 15:09:32.009787 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.009761 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 30 15:09:32.010299 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.010283 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 30 15:09:32.011962 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.011940 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 30 15:09:32.014718 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.013569 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-244.ec2.internal.18c719d329bb3a60 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-244.ec2.internal,UID:ip-10-0-142-244.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-244.ec2.internal,},FirstTimestamp:2026-07-30 15:09:32.004620896 +0000 UTC m=+0.345905664,LastTimestamp:2026-07-30 15:09:32.004620896 +0000 UTC m=+0.345905664,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-244.ec2.internal,}" Jul 30 15:09:32.017216 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.017200 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 30 15:09:32.017216 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.017210 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 30 15:09:32.017338 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.017264 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-244.ec2.internal\" not found" Jul 30 15:09:32.017855 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.017833 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 30 15:09:32.017957 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.017891 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 30 15:09:32.017957 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.017910 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 30 15:09:32.018057 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.018020 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 30 15:09:32.018057 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.018025 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 30 15:09:32.018263 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.018246 2579 factory.go:55] Registering systemd factory Jul 30 15:09:32.018263 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.018262 2579 factory.go:223] Registration of the systemd container factory successfully Jul 30 15:09:32.018489 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.018474 2579 factory.go:153] Registering CRI-O factory Jul 30 15:09:32.018489 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.018490 2579 factory.go:223] Registration of the crio container factory successfully Jul 30 15:09:32.018588 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.018530 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 30 15:09:32.018588 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.018550 2579 factory.go:103] Registering Raw factory Jul 30 15:09:32.018588 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.018559 2579 manager.go:1196] Started watching for new ooms in manager Jul 30 15:09:32.018911 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.018896 2579 manager.go:349] Starting recovery of all containers Jul 30 15:09:32.026815 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.026785 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-244.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 30 15:09:32.026926 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.026906 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 30 15:09:32.028434 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.028416 2579 manager.go:354] Recovery completed Jul 30 15:09:32.032940 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.032713 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 15:09:32.034743 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.034725 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasSufficientMemory" Jul 30 15:09:32.034818 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.034759 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 15:09:32.034818 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.034771 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasSufficientPID" Jul 30 15:09:32.035224 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.035208 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 30 15:09:32.035224 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.035221 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 15:09:32.035348 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.035222 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 30 15:09:32.035348 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.035294 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 30 15:09:32.035847 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.035834 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasSufficientMemory" Jul 30 15:09:32.035894 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.035857 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 15:09:32.035894 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.035866 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasSufficientPID" Jul 30 15:09:32.036617 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.036537 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-244.ec2.internal.18c719d32b86d8e7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-244.ec2.internal,UID:ip-10-0-142-244.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-244.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-244.ec2.internal,},FirstTimestamp:2026-07-30 15:09:32.034742503 +0000 UTC m=+0.376027273,LastTimestamp:2026-07-30 15:09:32.034742503 +0000 UTC m=+0.376027273,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-244.ec2.internal,}" Jul 30 15:09:32.037253 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.037240 2579 policy_none.go:49] "None policy: Start" Jul 30 15:09:32.037324 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.037258 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 30 15:09:32.037324 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.037271 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 30 15:09:32.038389 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.038376 2579 policy_none.go:47] "Start" Jul 30 15:09:32.039020 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.039002 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dp6wn" Jul 30 15:09:32.046864 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.046797 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-244.ec2.internal.18c719d32b872fcf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-244.ec2.internal,UID:ip-10-0-142-244.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-244.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-244.ec2.internal,},FirstTimestamp:2026-07-30 15:09:32.034764751 +0000 UTC m=+0.376049519,LastTimestamp:2026-07-30 15:09:32.034764751 +0000 UTC m=+0.376049519,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-244.ec2.internal,}" Jul 30 15:09:32.047333 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.047317 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dp6wn" Jul 30 15:09:32.067306 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.067185 2579 manager.go:337] "Starting Device Plugin manager" Jul 30 15:09:32.067395 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.067327 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 30 15:09:32.067395 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.067340 2579 server.go:87] "Starting device plugin registration server" Jul 30 15:09:32.067539 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.067518 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 30 15:09:32.067630 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.067532 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 30 15:09:32.067703 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.067681 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 30 15:09:32.067753 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.067745 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 30 15:09:32.067798 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.067753 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 30 15:09:32.069207 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.069160 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 30 15:09:32.069298 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.069210 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-244.ec2.internal\" not found" Jul 30 15:09:32.083601 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.083580 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 30 15:09:32.093175 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.084745 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 30 15:09:32.093175 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.084775 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 30 15:09:32.093175 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.084793 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 30 15:09:32.093175 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.084823 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 30 15:09:32.093175 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.086597 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 15:09:32.168568 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.168543 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 15:09:32.169282 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.169267 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasSufficientMemory" Jul 30 15:09:32.169360 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.169293 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 15:09:32.169360 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.169308 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasSufficientPID" Jul 30 15:09:32.169360 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.169328 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-244.ec2.internal" Jul 30 15:09:32.178657 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.178635 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-244.ec2.internal" Jul 30 15:09:32.178702 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.178662 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-244.ec2.internal\": node \"ip-10-0-142-244.ec2.internal\" not found" Jul 30 15:09:32.185746 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.185729 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-244.ec2.internal"] Jul 30 15:09:32.185814 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.185780 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 15:09:32.187077 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.187063 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasSufficientMemory" Jul 30 15:09:32.187139 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.187089 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 15:09:32.187139 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.187099 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasSufficientPID" Jul 30 15:09:32.188442 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.188430 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 15:09:32.188560 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.188544 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal" Jul 30 15:09:32.188600 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.188576 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 15:09:32.190055 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.189197 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasSufficientMemory" Jul 30 15:09:32.190055 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.189225 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 15:09:32.190055 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.189238 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasSufficientPID" Jul 30 15:09:32.190055 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.189279 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasSufficientMemory" Jul 30 15:09:32.190055 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.189300 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 15:09:32.190055 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.189312 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasSufficientPID" Jul 30 15:09:32.191000 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.190984 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.ec2.internal" Jul 30 15:09:32.191057 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.191012 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 15:09:32.191670 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.191628 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasSufficientMemory" Jul 30 15:09:32.191670 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.191668 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 15:09:32.191762 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.191678 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasSufficientPID" Jul 30 15:09:32.203931 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.203915 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-244.ec2.internal\" not found" Jul 30 15:09:32.218834 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.218814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf1e64c012dc1860c2dcd3c41af1093c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal\" (UID: \"cf1e64c012dc1860c2dcd3c41af1093c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal" Jul 30 15:09:32.218914 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.218838 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf1e64c012dc1860c2dcd3c41af1093c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal\" (UID: \"cf1e64c012dc1860c2dcd3c41af1093c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal" Jul 30 15:09:32.218914 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.218859 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/27a35d60ff6bc9476dbc920db8c64fe2-config\") pod \"kube-apiserver-proxy-ip-10-0-142-244.ec2.internal\" (UID: \"27a35d60ff6bc9476dbc920db8c64fe2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.ec2.internal" Jul 30 15:09:32.224305 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.224290 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-244.ec2.internal\" not found" node="ip-10-0-142-244.ec2.internal" Jul 30 15:09:32.228591 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.228577 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-244.ec2.internal\" not found" node="ip-10-0-142-244.ec2.internal" Jul 30 15:09:32.304097 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.304077 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-244.ec2.internal\" not found" Jul 30 15:09:32.319354 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.319337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf1e64c012dc1860c2dcd3c41af1093c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal\" (UID: \"cf1e64c012dc1860c2dcd3c41af1093c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal" Jul 30 15:09:32.319404 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.319364 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf1e64c012dc1860c2dcd3c41af1093c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal\" (UID: \"cf1e64c012dc1860c2dcd3c41af1093c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal" Jul 30 15:09:32.319404 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.319381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/27a35d60ff6bc9476dbc920db8c64fe2-config\") pod \"kube-apiserver-proxy-ip-10-0-142-244.ec2.internal\" (UID: \"27a35d60ff6bc9476dbc920db8c64fe2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.ec2.internal" Jul 30 15:09:32.319473 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.319429 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf1e64c012dc1860c2dcd3c41af1093c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal\" (UID: \"cf1e64c012dc1860c2dcd3c41af1093c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal" Jul 30 15:09:32.319473 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.319440 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf1e64c012dc1860c2dcd3c41af1093c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal\" (UID: \"cf1e64c012dc1860c2dcd3c41af1093c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal" Jul 30 15:09:32.319536 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.319429 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/27a35d60ff6bc9476dbc920db8c64fe2-config\") pod \"kube-apiserver-proxy-ip-10-0-142-244.ec2.internal\" (UID: \"27a35d60ff6bc9476dbc920db8c64fe2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.ec2.internal" Jul 30 15:09:32.404513 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.404482 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-244.ec2.internal\" not found" Jul 30 15:09:32.505225 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.505158 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-244.ec2.internal\" not found" Jul 30 15:09:32.527621 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.527603 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal" Jul 30 15:09:32.532897 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.532881 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.ec2.internal" Jul 30 15:09:32.605508 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.605476 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-244.ec2.internal\" not found" Jul 30 15:09:32.706018 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.705998 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-244.ec2.internal\" not found" Jul 30 15:09:32.806559 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.806529 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-244.ec2.internal\" not found" Jul 30 15:09:32.907123 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:32.907087 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-244.ec2.internal\" not found" Jul 30 15:09:32.959462 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.959443 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 30 15:09:32.959834 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:32.959574 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 30 15:09:33.007999 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:33.007969 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-244.ec2.internal\" not found" Jul 30 15:09:33.052432 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:33.052399 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 15:04:33 +0000 UTC" deadline="2028-01-20 18:47:40.617686004 +0000 UTC" Jul 30 15:09:33.052432 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:33.052430 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12939h38m7.565260157s" Jul 30 15:09:33.108713 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:33.108650 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-244.ec2.internal\" not found" Jul 30 15:09:33.125916 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:33.125892 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf1e64c012dc1860c2dcd3c41af1093c.slice/crio-93b9209c94b6766020e536298c2d313a32632560db252b1da24f41c166137e6b WatchSource:0}: Error finding container 93b9209c94b6766020e536298c2d313a32632560db252b1da24f41c166137e6b: Status 404 returned error can't find the container with id 93b9209c94b6766020e536298c2d313a32632560db252b1da24f41c166137e6b Jul 30 15:09:33.129619 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:33.129604 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 15:09:33.159221 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:33.159199 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 15:09:33.189352 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:33.189336 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 15:09:33.190395 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:33.190379 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasSufficientMemory" Jul 30 15:09:33.190473 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:33.190409 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 15:09:33.190473 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:33.190420 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeHasSufficientPID" Jul 30 15:09:34.315920 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:34.315897 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-244.ec2.internal\" not found" Jul 30 15:09:34.355273 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:34.355256 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 15:09:34.416806 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:34.416759 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-244.ec2.internal\" not found" Jul 30 15:09:34.429548 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:34.429530 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 15:09:34.524720 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:34.524669 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal" Jul 30 15:09:34.539532 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:34.539516 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 15:09:34.540914 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:34.540902 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.ec2.internal" Jul 30 15:09:34.550328 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:34.550313 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 15:09:34.848725 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:34.848692 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a35d60ff6bc9476dbc920db8c64fe2.slice/crio-c1baffadaa387c5e503110fb6f995abde5b069c1766fcfa78805b18dede825d3 WatchSource:0}: Error finding container c1baffadaa387c5e503110fb6f995abde5b069c1766fcfa78805b18dede825d3: Status 404 returned error can't find the container with id c1baffadaa387c5e503110fb6f995abde5b069c1766fcfa78805b18dede825d3 Jul 30 15:09:35.101256 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.101188 2579 apiserver.go:52] "Watching apiserver" Jul 30 15:09:35.114377 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.114352 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 30 15:09:35.114738 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.114713 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-244.ec2.internal","openshift-cluster-node-tuning-operator/tuned-zjqcf","openshift-dns/node-resolver-cvbfc","openshift-image-registry/node-ca-g8lsb","openshift-multus/network-metrics-daemon-rsf9b","openshift-network-diagnostics/network-check-target-pc6xx","openshift-ovn-kubernetes/ovnkube-node-2pbpn","kube-system/konnectivity-agent-xj2rx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal","openshift-multus/multus-additional-cni-plugins-qbrvz","openshift-multus/multus-q574q","openshift-network-operator/iptables-alerter-grsr9","kube-system/global-pull-secret-syncer-qh4bp"] Jul 30 15:09:35.116662 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.116641 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.118260 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.118237 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-grsr9" Jul 30 15:09:35.119521 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.119493 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cvbfc" Jul 30 15:09:35.119654 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.119634 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9wgz9\"" Jul 30 15:09:35.120747 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.120727 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 30 15:09:35.120909 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.120877 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 30 15:09:35.121037 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.121021 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 30 15:09:35.121152 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.121081 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g8lsb" Jul 30 15:09:35.121400 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.121379 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 30 15:09:35.122273 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.122228 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:35.122353 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.122298 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 30 15:09:35.122353 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.122299 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsf9b" podUID="a8cf4460-ac48-44e7-9a6e-64ff431ca403" Jul 30 15:09:35.122477 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.122460 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 30 15:09:35.122590 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.122572 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 30 15:09:35.122760 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.122744 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 30 15:09:35.122858 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.122810 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 30 15:09:35.122919 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.122863 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zxdsg\"" Jul 30 15:09:35.122970 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.122947 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 30 15:09:35.123017 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.123006 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j75hw\"" Jul 30 15:09:35.123630 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.123613 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:35.123741 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.123673 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 30 15:09:35.123741 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.123671 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc6xx" podUID="496d6e44-f8b2-447b-b325-9b9be2c0e935" Jul 30 15:09:35.124053 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.124034 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 30 15:09:35.124128 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.124055 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 30 15:09:35.124128 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.124113 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kpcjs\"" Jul 30 15:09:35.124222 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.124204 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 30 15:09:35.124312 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.124296 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 30 15:09:35.125329 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.125278 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.126997 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.126976 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:35.127138 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.127066 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4bp" podUID="113d506f-fc3f-4ea8-9eb2-98a534c8b5db" Jul 30 15:09:35.127597 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.127581 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 30 15:09:35.127781 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.127760 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 30 15:09:35.127875 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.127855 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 30 15:09:35.127875 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.127871 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-phm5d\"" Jul 30 15:09:35.128361 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.128343 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.129728 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.129712 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xj2rx" Jul 30 15:09:35.131010 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.130992 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 30 15:09:35.131355 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.131060 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 30 15:09:35.131355 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.131137 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p4j4q\"" Jul 30 15:09:35.131355 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.131282 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.132522 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.132340 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cvkbc\"" Jul 30 15:09:35.132522 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.132346 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 30 15:09:35.132522 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.132449 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 30 15:09:35.133510 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.133490 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q574q" Jul 30 15:09:35.134388 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.134368 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 30 15:09:35.134484 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.134461 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b9fwp\"" Jul 30 15:09:35.134574 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.134560 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 30 15:09:35.134627 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.134600 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 30 15:09:35.134772 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.134754 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 30 15:09:35.134971 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.134953 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 30 15:09:35.135179 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqzr6\" (UniqueName: \"kubernetes.io/projected/01d7c93b-7592-4f23-aea2-96edecd05fe9-kube-api-access-mqzr6\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.135253 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135224 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c35ba36d-8199-471f-99df-70cc5ba0c05e-hosts-file\") pod \"node-resolver-cvbfc\" (UID: \"c35ba36d-8199-471f-99df-70cc5ba0c05e\") " pod="openshift-dns/node-resolver-cvbfc" Jul 30 15:09:35.135302 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135263 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvg49\" (UniqueName: \"kubernetes.io/projected/c35ba36d-8199-471f-99df-70cc5ba0c05e-kube-api-access-tvg49\") pod \"node-resolver-cvbfc\" (UID: \"c35ba36d-8199-471f-99df-70cc5ba0c05e\") " pod="openshift-dns/node-resolver-cvbfc" Jul 30 15:09:35.135361 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135313 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhxfc\" (UniqueName: \"kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc\") pod \"network-check-target-pc6xx\" (UID: \"496d6e44-f8b2-447b-b325-9b9be2c0e935\") " pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:35.135414 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/adde37e6-1714-44ed-bb3d-57d3afcd1337-kubelet-dir\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.135414 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135406 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/adde37e6-1714-44ed-bb3d-57d3afcd1337-sys-fs\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.135510 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135440 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ec9478e9-d9e0-4ea7-9412-e36fc1a305c7-serviceca\") pod \"node-ca-g8lsb\" (UID: \"ec9478e9-d9e0-4ea7-9412-e36fc1a305c7\") " pod="openshift-image-registry/node-ca-g8lsb" Jul 30 15:09:35.135510 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135465 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrkh8\" (UniqueName: \"kubernetes.io/projected/ec9478e9-d9e0-4ea7-9412-e36fc1a305c7-kube-api-access-nrkh8\") pod \"node-ca-g8lsb\" (UID: \"ec9478e9-d9e0-4ea7-9412-e36fc1a305c7\") " pod="openshift-image-registry/node-ca-g8lsb" Jul 30 15:09:35.135510 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135493 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db073d9a-53e8-4a4b-bc78-be972bc96d24-host-slash\") pod \"iptables-alerter-grsr9\" (UID: \"db073d9a-53e8-4a4b-bc78-be972bc96d24\") " pod="openshift-network-operator/iptables-alerter-grsr9" Jul 30 15:09:35.135653 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135521 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/adde37e6-1714-44ed-bb3d-57d3afcd1337-device-dir\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.135653 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-run-netns\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.135653 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135590 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-run-ovn-kubernetes\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.135653 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135628 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qshht\" (UniqueName: \"kubernetes.io/projected/db073d9a-53e8-4a4b-bc78-be972bc96d24-kube-api-access-qshht\") pod \"iptables-alerter-grsr9\" (UID: \"db073d9a-53e8-4a4b-bc78-be972bc96d24\") " pod="openshift-network-operator/iptables-alerter-grsr9" Jul 30 15:09:35.135873 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135659 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c35ba36d-8199-471f-99df-70cc5ba0c05e-tmp-dir\") pod \"node-resolver-cvbfc\" (UID: \"c35ba36d-8199-471f-99df-70cc5ba0c05e\") " pod="openshift-dns/node-resolver-cvbfc" Jul 30 15:09:35.135873 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135705 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-systemd-units\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.135873 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-run-openvswitch\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.135873 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.135873 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/01d7c93b-7592-4f23-aea2-96edecd05fe9-ovnkube-config\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.136106 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135890 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/01d7c93b-7592-4f23-aea2-96edecd05fe9-ovnkube-script-lib\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.136106 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135919 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-kubelet\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.136106 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.135981 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-var-lib-openvswitch\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.136106 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-log-socket\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.136106 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136035 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-cni-bin\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.136106 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136065 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/01d7c93b-7592-4f23-aea2-96edecd05fe9-env-overrides\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.136106 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136094 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/01d7c93b-7592-4f23-aea2-96edecd05fe9-ovn-node-metrics-cert\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.136441 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136151 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/adde37e6-1714-44ed-bb3d-57d3afcd1337-etc-selinux\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.136441 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136205 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-etc-openvswitch\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.136441 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136216 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 30 15:09:35.136441 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136232 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-run-ovn\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.136441 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136263 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-node-log\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.136441 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kwzf\" (UniqueName: \"kubernetes.io/projected/adde37e6-1714-44ed-bb3d-57d3afcd1337-kube-api-access-8kwzf\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.136441 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-697zs\" (UniqueName: \"kubernetes.io/projected/a8cf4460-ac48-44e7-9a6e-64ff431ca403-kube-api-access-697zs\") pod \"network-metrics-daemon-rsf9b\" (UID: \"a8cf4460-ac48-44e7-9a6e-64ff431ca403\") " pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:35.136441 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136408 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db073d9a-53e8-4a4b-bc78-be972bc96d24-iptables-alerter-script\") pod \"iptables-alerter-grsr9\" (UID: \"db073d9a-53e8-4a4b-bc78-be972bc96d24\") " pod="openshift-network-operator/iptables-alerter-grsr9" Jul 30 15:09:35.136837 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/adde37e6-1714-44ed-bb3d-57d3afcd1337-socket-dir\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.136837 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/adde37e6-1714-44ed-bb3d-57d3afcd1337-registration-dir\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.136837 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136561 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs\") pod \"network-metrics-daemon-rsf9b\" (UID: \"a8cf4460-ac48-44e7-9a6e-64ff431ca403\") " pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:35.136837 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136599 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-slash\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.136837 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136628 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-run-systemd\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.136837 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-cni-netd\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.136837 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.136707 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec9478e9-d9e0-4ea7-9412-e36fc1a305c7-host\") pod \"node-ca-g8lsb\" (UID: \"ec9478e9-d9e0-4ea7-9412-e36fc1a305c7\") " pod="openshift-image-registry/node-ca-g8lsb" Jul 30 15:09:35.137619 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.137457 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-667p6\"" Jul 30 15:09:35.137699 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.137669 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 15:09:35.162026 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.162004 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qkhrt" Jul 30 15:09:35.169404 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.169388 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qkhrt" Jul 30 15:09:35.195692 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.195650 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.ec2.internal" event={"ID":"27a35d60ff6bc9476dbc920db8c64fe2","Type":"ContainerStarted","Data":"c1baffadaa387c5e503110fb6f995abde5b069c1766fcfa78805b18dede825d3"} Jul 30 15:09:35.196689 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.196669 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal" event={"ID":"cf1e64c012dc1860c2dcd3c41af1093c","Type":"ContainerStarted","Data":"93b9209c94b6766020e536298c2d313a32632560db252b1da24f41c166137e6b"} Jul 30 15:09:35.225656 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.225632 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 30 15:09:35.237860 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.237837 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/693f4057-2498-479f-912c-c1f3f77aefab-etc-tuned\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.237968 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.237885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-node-log\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.237968 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.237915 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-system-cni-dir\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.237968 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.237921 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-node-log\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.237968 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.237941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-multus-socket-dir-parent\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.237968 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.237966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-host-run-k8s-cni-cncf-io\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.238211 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.237990 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8k84\" (UniqueName: \"kubernetes.io/projected/e04c52a8-91f2-49a2-9e17-c015e0c44d36-kube-api-access-l8k84\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.238211 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238018 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db073d9a-53e8-4a4b-bc78-be972bc96d24-iptables-alerter-script\") pod \"iptables-alerter-grsr9\" (UID: \"db073d9a-53e8-4a4b-bc78-be972bc96d24\") " pod="openshift-network-operator/iptables-alerter-grsr9" Jul 30 15:09:35.238211 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/adde37e6-1714-44ed-bb3d-57d3afcd1337-socket-dir\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.238211 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238070 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/adde37e6-1714-44ed-bb3d-57d3afcd1337-registration-dir\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.238211 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238093 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd846133-3b4c-459a-abd2-58609374a7e8-agent-certs\") pod \"konnectivity-agent-xj2rx\" (UID: \"cd846133-3b4c-459a-abd2-58609374a7e8\") " pod="kube-system/konnectivity-agent-xj2rx" Jul 30 15:09:35.238211 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-cni-netd\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.238211 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238164 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c66d9\" (UniqueName: \"kubernetes.io/projected/693f4057-2498-479f-912c-c1f3f77aefab-kube-api-access-c66d9\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.238211 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238172 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/adde37e6-1714-44ed-bb3d-57d3afcd1337-registration-dir\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.238211 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-host-run-netns\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.238211 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238195 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-cni-netd\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.238211 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238207 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/adde37e6-1714-44ed-bb3d-57d3afcd1337-socket-dir\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.238815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e04c52a8-91f2-49a2-9e17-c015e0c44d36-system-cni-dir\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.238815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqzr6\" (UniqueName: \"kubernetes.io/projected/01d7c93b-7592-4f23-aea2-96edecd05fe9-kube-api-access-mqzr6\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.238815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c35ba36d-8199-471f-99df-70cc5ba0c05e-hosts-file\") pod \"node-resolver-cvbfc\" (UID: \"c35ba36d-8199-471f-99df-70cc5ba0c05e\") " pod="openshift-dns/node-resolver-cvbfc" Jul 30 15:09:35.238815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvg49\" (UniqueName: \"kubernetes.io/projected/c35ba36d-8199-471f-99df-70cc5ba0c05e-kube-api-access-tvg49\") pod \"node-resolver-cvbfc\" (UID: \"c35ba36d-8199-471f-99df-70cc5ba0c05e\") " pod="openshift-dns/node-resolver-cvbfc" Jul 30 15:09:35.238815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/adde37e6-1714-44ed-bb3d-57d3afcd1337-sys-fs\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.238815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238350 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c35ba36d-8199-471f-99df-70cc5ba0c05e-hosts-file\") pod \"node-resolver-cvbfc\" (UID: \"c35ba36d-8199-471f-99df-70cc5ba0c05e\") " pod="openshift-dns/node-resolver-cvbfc" Jul 30 15:09:35.238815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-sys\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.238815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238398 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-etc-sysctl-conf\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.238815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret\") pod \"global-pull-secret-syncer-qh4bp\" (UID: \"113d506f-fc3f-4ea8-9eb2-98a534c8b5db\") " pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:35.238815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-run-ovn-kubernetes\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.238815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238472 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qshht\" (UniqueName: \"kubernetes.io/projected/db073d9a-53e8-4a4b-bc78-be972bc96d24-kube-api-access-qshht\") pod \"iptables-alerter-grsr9\" (UID: \"db073d9a-53e8-4a4b-bc78-be972bc96d24\") " pod="openshift-network-operator/iptables-alerter-grsr9" Jul 30 15:09:35.238815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238496 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c35ba36d-8199-471f-99df-70cc5ba0c05e-tmp-dir\") pod \"node-resolver-cvbfc\" (UID: \"c35ba36d-8199-471f-99df-70cc5ba0c05e\") " pod="openshift-dns/node-resolver-cvbfc" Jul 30 15:09:35.238815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238519 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3999367-09f0-4979-8867-d16af22db8f6-cni-binary-copy\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.238815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238544 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-host-var-lib-cni-bin\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.238815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238552 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/adde37e6-1714-44ed-bb3d-57d3afcd1337-sys-fs\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.238815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-hostroot\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.238815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238576 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db073d9a-53e8-4a4b-bc78-be972bc96d24-iptables-alerter-script\") pod \"iptables-alerter-grsr9\" (UID: \"db073d9a-53e8-4a4b-bc78-be972bc96d24\") " pod="openshift-network-operator/iptables-alerter-grsr9" Jul 30 15:09:35.239562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3999367-09f0-4979-8867-d16af22db8f6-multus-daemon-config\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.239562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238618 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-host-run-multus-certs\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.239562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-systemd-units\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.239562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238651 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-run-ovn-kubernetes\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.239562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238666 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/01d7c93b-7592-4f23-aea2-96edecd05fe9-ovnkube-config\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.239562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238702 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-os-release\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.239562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238733 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e04c52a8-91f2-49a2-9e17-c015e0c44d36-cni-binary-copy\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.239562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-log-socket\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.239562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238788 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/01d7c93b-7592-4f23-aea2-96edecd05fe9-env-overrides\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.239562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/adde37e6-1714-44ed-bb3d-57d3afcd1337-etc-selinux\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.239562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238841 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-log-socket\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.239562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238856 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-etc-modprobe-d\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.239562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-lib-modules\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.239562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238900 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c35ba36d-8199-471f-99df-70cc5ba0c05e-tmp-dir\") pod \"node-resolver-cvbfc\" (UID: \"c35ba36d-8199-471f-99df-70cc5ba0c05e\") " pod="openshift-dns/node-resolver-cvbfc" Jul 30 15:09:35.239562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-cnibin\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.239562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238961 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/adde37e6-1714-44ed-bb3d-57d3afcd1337-etc-selinux\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.239562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-etc-openvswitch\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.240319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.238997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-run-ovn\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.240319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239014 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-systemd-units\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.240319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kwzf\" (UniqueName: \"kubernetes.io/projected/adde37e6-1714-44ed-bb3d-57d3afcd1337-kube-api-access-8kwzf\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.240319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239073 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-etc-openvswitch\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.240319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239097 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-697zs\" (UniqueName: \"kubernetes.io/projected/a8cf4460-ac48-44e7-9a6e-64ff431ca403-kube-api-access-697zs\") pod \"network-metrics-daemon-rsf9b\" (UID: \"a8cf4460-ac48-44e7-9a6e-64ff431ca403\") " pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:35.240319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239123 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-etc-sysctl-d\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.240319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239147 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs\") pod \"network-metrics-daemon-rsf9b\" (UID: \"a8cf4460-ac48-44e7-9a6e-64ff431ca403\") " pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:35.240319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239170 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-etc-kubernetes\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.240319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239191 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-run\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.240319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239212 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/693f4057-2498-479f-912c-c1f3f77aefab-tmp\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.240319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239233 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-kubelet-config\") pod \"global-pull-secret-syncer-qh4bp\" (UID: \"113d506f-fc3f-4ea8-9eb2-98a534c8b5db\") " pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:35.240319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239240 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/01d7c93b-7592-4f23-aea2-96edecd05fe9-ovnkube-config\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.240319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239255 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e04c52a8-91f2-49a2-9e17-c015e0c44d36-cnibin\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.240319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239293 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-slash\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.240319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239314 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-run-ovn\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.240319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239304 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/01d7c93b-7592-4f23-aea2-96edecd05fe9-env-overrides\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.240319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239330 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-run-systemd\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.241017 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239360 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-slash\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.241017 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec9478e9-d9e0-4ea7-9412-e36fc1a305c7-host\") pod \"node-ca-g8lsb\" (UID: \"ec9478e9-d9e0-4ea7-9412-e36fc1a305c7\") " pod="openshift-image-registry/node-ca-g8lsb" Jul 30 15:09:35.241017 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-etc-systemd\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.241017 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239430 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec9478e9-d9e0-4ea7-9412-e36fc1a305c7-host\") pod \"node-ca-g8lsb\" (UID: \"ec9478e9-d9e0-4ea7-9412-e36fc1a305c7\") " pod="openshift-image-registry/node-ca-g8lsb" Jul 30 15:09:35.241017 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-host\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.241017 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239470 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-run-systemd\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.241017 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-host-var-lib-cni-multus\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.241017 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.239544 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 15:09:35.241017 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239568 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-etc-kubernetes\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.241017 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239596 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxfc\" (UniqueName: \"kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc\") pod \"network-check-target-pc6xx\" (UID: \"496d6e44-f8b2-447b-b325-9b9be2c0e935\") " pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:35.241017 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.239637 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs podName:a8cf4460-ac48-44e7-9a6e-64ff431ca403 nodeName:}" failed. No retries permitted until 2026-07-30 15:09:35.739604343 +0000 UTC m=+2.974735187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs") pod "network-metrics-daemon-rsf9b" (UID: "a8cf4460-ac48-44e7-9a6e-64ff431ca403") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 15:09:35.241017 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239681 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/adde37e6-1714-44ed-bb3d-57d3afcd1337-kubelet-dir\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.241017 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ec9478e9-d9e0-4ea7-9412-e36fc1a305c7-serviceca\") pod \"node-ca-g8lsb\" (UID: \"ec9478e9-d9e0-4ea7-9412-e36fc1a305c7\") " pod="openshift-image-registry/node-ca-g8lsb" Jul 30 15:09:35.241017 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239747 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/adde37e6-1714-44ed-bb3d-57d3afcd1337-kubelet-dir\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.241017 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239739 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkh8\" (UniqueName: \"kubernetes.io/projected/ec9478e9-d9e0-4ea7-9412-e36fc1a305c7-kube-api-access-nrkh8\") pod \"node-ca-g8lsb\" (UID: \"ec9478e9-d9e0-4ea7-9412-e36fc1a305c7\") " pod="openshift-image-registry/node-ca-g8lsb" Jul 30 15:09:35.241017 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239807 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-etc-sysconfig\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.241017 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-multus-cni-dir\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.241643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239853 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-host-var-lib-kubelet\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.241643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239877 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcwqt\" (UniqueName: \"kubernetes.io/projected/f3999367-09f0-4979-8867-d16af22db8f6-kube-api-access-qcwqt\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.241643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239908 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db073d9a-53e8-4a4b-bc78-be972bc96d24-host-slash\") pod \"iptables-alerter-grsr9\" (UID: \"db073d9a-53e8-4a4b-bc78-be972bc96d24\") " pod="openshift-network-operator/iptables-alerter-grsr9" Jul 30 15:09:35.241643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239933 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/adde37e6-1714-44ed-bb3d-57d3afcd1337-device-dir\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.241643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239957 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-dbus\") pod \"global-pull-secret-syncer-qh4bp\" (UID: \"113d506f-fc3f-4ea8-9eb2-98a534c8b5db\") " pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:35.241643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239982 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd846133-3b4c-459a-abd2-58609374a7e8-konnectivity-ca\") pod \"konnectivity-agent-xj2rx\" (UID: \"cd846133-3b4c-459a-abd2-58609374a7e8\") " pod="kube-system/konnectivity-agent-xj2rx" Jul 30 15:09:35.241643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.239998 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db073d9a-53e8-4a4b-bc78-be972bc96d24-host-slash\") pod \"iptables-alerter-grsr9\" (UID: \"db073d9a-53e8-4a4b-bc78-be972bc96d24\") " pod="openshift-network-operator/iptables-alerter-grsr9" Jul 30 15:09:35.241643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240007 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e04c52a8-91f2-49a2-9e17-c015e0c44d36-os-release\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.241643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240017 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/adde37e6-1714-44ed-bb3d-57d3afcd1337-device-dir\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.241643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240044 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e04c52a8-91f2-49a2-9e17-c015e0c44d36-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.241643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240072 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e04c52a8-91f2-49a2-9e17-c015e0c44d36-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.241643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240098 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-run-netns\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.241643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240108 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ec9478e9-d9e0-4ea7-9412-e36fc1a305c7-serviceca\") pod \"node-ca-g8lsb\" (UID: \"ec9478e9-d9e0-4ea7-9412-e36fc1a305c7\") " pod="openshift-image-registry/node-ca-g8lsb" Jul 30 15:09:35.241643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240133 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-run-openvswitch\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.241643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240167 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-run-openvswitch\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.241643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240179 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.241643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240203 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-run-netns\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.242266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/01d7c93b-7592-4f23-aea2-96edecd05fe9-ovnkube-script-lib\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.242266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240241 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.242266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-kubelet\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.242266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-var-lib-openvswitch\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.242266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-cni-bin\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.242266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240456 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-var-lib-openvswitch\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.242266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/01d7c93b-7592-4f23-aea2-96edecd05fe9-ovn-node-metrics-cert\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.242266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240382 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-kubelet\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.242266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240495 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-multus-conf-dir\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.242266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240502 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01d7c93b-7592-4f23-aea2-96edecd05fe9-host-cni-bin\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.242266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240520 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e04c52a8-91f2-49a2-9e17-c015e0c44d36-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.242266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240554 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-var-lib-kubelet\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.242266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240763 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/01d7c93b-7592-4f23-aea2-96edecd05fe9-ovnkube-script-lib\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.242266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.240785 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 30 15:09:35.244134 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.244114 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/01d7c93b-7592-4f23-aea2-96edecd05fe9-ovn-node-metrics-cert\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.257950 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.257928 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 15:09:35.258033 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.257956 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 15:09:35.258033 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.257970 2579 projected.go:196] Error preparing data for projected volume kube-api-access-xhxfc for pod openshift-network-diagnostics/network-check-target-pc6xx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 15:09:35.258033 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.258028 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc podName:496d6e44-f8b2-447b-b325-9b9be2c0e935 nodeName:}" failed. No retries permitted until 2026-07-30 15:09:35.758012408 +0000 UTC m=+2.993143299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xhxfc" (UniqueName: "kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc") pod "network-check-target-pc6xx" (UID: "496d6e44-f8b2-447b-b325-9b9be2c0e935") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 15:09:35.260599 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.260539 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvg49\" (UniqueName: \"kubernetes.io/projected/c35ba36d-8199-471f-99df-70cc5ba0c05e-kube-api-access-tvg49\") pod \"node-resolver-cvbfc\" (UID: \"c35ba36d-8199-471f-99df-70cc5ba0c05e\") " pod="openshift-dns/node-resolver-cvbfc" Jul 30 15:09:35.260817 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.260779 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kwzf\" (UniqueName: \"kubernetes.io/projected/adde37e6-1714-44ed-bb3d-57d3afcd1337-kube-api-access-8kwzf\") pod \"aws-ebs-csi-driver-node-57rvj\" (UID: \"adde37e6-1714-44ed-bb3d-57d3afcd1337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.261408 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.261385 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-697zs\" (UniqueName: \"kubernetes.io/projected/a8cf4460-ac48-44e7-9a6e-64ff431ca403-kube-api-access-697zs\") pod \"network-metrics-daemon-rsf9b\" (UID: \"a8cf4460-ac48-44e7-9a6e-64ff431ca403\") " pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:35.261818 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.261781 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qshht\" (UniqueName: \"kubernetes.io/projected/db073d9a-53e8-4a4b-bc78-be972bc96d24-kube-api-access-qshht\") pod \"iptables-alerter-grsr9\" (UID: \"db073d9a-53e8-4a4b-bc78-be972bc96d24\") " pod="openshift-network-operator/iptables-alerter-grsr9" Jul 30 15:09:35.262337 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.262290 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkh8\" (UniqueName: \"kubernetes.io/projected/ec9478e9-d9e0-4ea7-9412-e36fc1a305c7-kube-api-access-nrkh8\") pod \"node-ca-g8lsb\" (UID: \"ec9478e9-d9e0-4ea7-9412-e36fc1a305c7\") " pod="openshift-image-registry/node-ca-g8lsb" Jul 30 15:09:35.262486 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.262447 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqzr6\" (UniqueName: \"kubernetes.io/projected/01d7c93b-7592-4f23-aea2-96edecd05fe9-kube-api-access-mqzr6\") pod \"ovnkube-node-2pbpn\" (UID: \"01d7c93b-7592-4f23-aea2-96edecd05fe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.341146 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-etc-sysctl-d\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.341146 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341157 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-etc-kubernetes\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.341365 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341177 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-run\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.341365 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341251 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-etc-kubernetes\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.341365 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/693f4057-2498-479f-912c-c1f3f77aefab-tmp\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.341365 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341286 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-etc-sysctl-d\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.341365 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341314 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-kubelet-config\") pod \"global-pull-secret-syncer-qh4bp\" (UID: \"113d506f-fc3f-4ea8-9eb2-98a534c8b5db\") " pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:35.341365 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341284 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-run\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.341365 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e04c52a8-91f2-49a2-9e17-c015e0c44d36-cnibin\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.341696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341395 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e04c52a8-91f2-49a2-9e17-c015e0c44d36-cnibin\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.341696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341401 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-kubelet-config\") pod \"global-pull-secret-syncer-qh4bp\" (UID: \"113d506f-fc3f-4ea8-9eb2-98a534c8b5db\") " pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:35.341696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341426 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-etc-systemd\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.341696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-host\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.341696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341469 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-host-var-lib-cni-multus\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.341696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-etc-kubernetes\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.341696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341518 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-etc-systemd\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.341696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-etc-sysconfig\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.341696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341547 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-host\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.341696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-multus-cni-dir\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.341696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-host-var-lib-kubelet\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.341696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341590 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-etc-kubernetes\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.341696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341561 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-host-var-lib-cni-multus\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.341696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcwqt\" (UniqueName: \"kubernetes.io/projected/f3999367-09f0-4979-8867-d16af22db8f6-kube-api-access-qcwqt\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.341696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341614 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-etc-sysconfig\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.341696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341632 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-dbus\") pod \"global-pull-secret-syncer-qh4bp\" (UID: \"113d506f-fc3f-4ea8-9eb2-98a534c8b5db\") " pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:35.341696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341640 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-host-var-lib-kubelet\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.341696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd846133-3b4c-459a-abd2-58609374a7e8-konnectivity-ca\") pod \"konnectivity-agent-xj2rx\" (UID: \"cd846133-3b4c-459a-abd2-58609374a7e8\") " pod="kube-system/konnectivity-agent-xj2rx" Jul 30 15:09:35.342553 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e04c52a8-91f2-49a2-9e17-c015e0c44d36-os-release\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.342553 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e04c52a8-91f2-49a2-9e17-c015e0c44d36-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.342553 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341719 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e04c52a8-91f2-49a2-9e17-c015e0c44d36-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.342553 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341736 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-dbus\") pod \"global-pull-secret-syncer-qh4bp\" (UID: \"113d506f-fc3f-4ea8-9eb2-98a534c8b5db\") " pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:35.342553 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341737 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-multus-cni-dir\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.342553 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341753 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-multus-conf-dir\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.342553 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e04c52a8-91f2-49a2-9e17-c015e0c44d36-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.342553 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341812 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e04c52a8-91f2-49a2-9e17-c015e0c44d36-os-release\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.342553 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-var-lib-kubelet\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.342553 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/693f4057-2498-479f-912c-c1f3f77aefab-etc-tuned\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.342553 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341876 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-system-cni-dir\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.342553 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341878 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e04c52a8-91f2-49a2-9e17-c015e0c44d36-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.342553 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341899 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-multus-socket-dir-parent\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.342553 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.341952 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-multus-socket-dir-parent\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.342553 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342002 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-var-lib-kubelet\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.342553 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342120 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-multus-conf-dir\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.342553 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342138 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-system-cni-dir\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.343347 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-host-run-k8s-cni-cncf-io\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.343347 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342182 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-host-run-k8s-cni-cncf-io\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.343347 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8k84\" (UniqueName: \"kubernetes.io/projected/e04c52a8-91f2-49a2-9e17-c015e0c44d36-kube-api-access-l8k84\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.343347 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342221 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd846133-3b4c-459a-abd2-58609374a7e8-agent-certs\") pod \"konnectivity-agent-xj2rx\" (UID: \"cd846133-3b4c-459a-abd2-58609374a7e8\") " pod="kube-system/konnectivity-agent-xj2rx" Jul 30 15:09:35.343347 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342250 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c66d9\" (UniqueName: \"kubernetes.io/projected/693f4057-2498-479f-912c-c1f3f77aefab-kube-api-access-c66d9\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.343347 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342275 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-host-run-netns\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.343347 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e04c52a8-91f2-49a2-9e17-c015e0c44d36-system-cni-dir\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.343347 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-sys\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.343347 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342351 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd846133-3b4c-459a-abd2-58609374a7e8-konnectivity-ca\") pod \"konnectivity-agent-xj2rx\" (UID: \"cd846133-3b4c-459a-abd2-58609374a7e8\") " pod="kube-system/konnectivity-agent-xj2rx" Jul 30 15:09:35.343347 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-etc-sysctl-conf\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.343347 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342389 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret\") pod \"global-pull-secret-syncer-qh4bp\" (UID: \"113d506f-fc3f-4ea8-9eb2-98a534c8b5db\") " pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:35.343347 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3999367-09f0-4979-8867-d16af22db8f6-cni-binary-copy\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.343347 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-host-var-lib-cni-bin\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.343347 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342463 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-hostroot\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.343347 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342483 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3999367-09f0-4979-8867-d16af22db8f6-multus-daemon-config\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.343347 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342505 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-host-run-multus-certs\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.343347 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342510 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-host-run-netns\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.344214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342416 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e04c52a8-91f2-49a2-9e17-c015e0c44d36-system-cni-dir\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.344214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-os-release\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.344214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342581 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-sys\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.344214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342604 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-os-release\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.344214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342617 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-hostroot\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.344214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e04c52a8-91f2-49a2-9e17-c015e0c44d36-cni-binary-copy\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.344214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342669 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-etc-modprobe-d\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.344214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342692 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-lib-modules\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.344214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342697 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-etc-sysctl-conf\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.344214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342740 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-host-var-lib-cni-bin\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.344214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342815 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-lib-modules\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.344214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342844 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-host-run-multus-certs\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.344214 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.342892 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 15:09:35.344214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.342937 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/693f4057-2498-479f-912c-c1f3f77aefab-etc-modprobe-d\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.344214 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.342987 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret podName:113d506f-fc3f-4ea8-9eb2-98a534c8b5db nodeName:}" failed. No retries permitted until 2026-07-30 15:09:35.842968795 +0000 UTC m=+3.078099629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret") pod "global-pull-secret-syncer-qh4bp" (UID: "113d506f-fc3f-4ea8-9eb2-98a534c8b5db") : object "kube-system"/"original-pull-secret" not registered Jul 30 15:09:35.344214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.343031 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e04c52a8-91f2-49a2-9e17-c015e0c44d36-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.344214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.343180 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3999367-09f0-4979-8867-d16af22db8f6-multus-daemon-config\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.345032 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.343220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-cnibin\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.345032 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.343288 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e04c52a8-91f2-49a2-9e17-c015e0c44d36-cni-binary-copy\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.345032 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.343297 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3999367-09f0-4979-8867-d16af22db8f6-cnibin\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.345032 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.343389 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e04c52a8-91f2-49a2-9e17-c015e0c44d36-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.345032 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.343423 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3999367-09f0-4979-8867-d16af22db8f6-cni-binary-copy\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.345032 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.344002 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/693f4057-2498-479f-912c-c1f3f77aefab-tmp\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.345306 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.345045 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/693f4057-2498-479f-912c-c1f3f77aefab-etc-tuned\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.345306 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.345155 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd846133-3b4c-459a-abd2-58609374a7e8-agent-certs\") pod \"konnectivity-agent-xj2rx\" (UID: \"cd846133-3b4c-459a-abd2-58609374a7e8\") " pod="kube-system/konnectivity-agent-xj2rx" Jul 30 15:09:35.352056 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.352001 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8k84\" (UniqueName: \"kubernetes.io/projected/e04c52a8-91f2-49a2-9e17-c015e0c44d36-kube-api-access-l8k84\") pod \"multus-additional-cni-plugins-qbrvz\" (UID: \"e04c52a8-91f2-49a2-9e17-c015e0c44d36\") " pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.352266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.352248 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c66d9\" (UniqueName: \"kubernetes.io/projected/693f4057-2498-479f-912c-c1f3f77aefab-kube-api-access-c66d9\") pod \"tuned-zjqcf\" (UID: \"693f4057-2498-479f-912c-c1f3f77aefab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.352326 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.352308 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcwqt\" (UniqueName: \"kubernetes.io/projected/f3999367-09f0-4979-8867-d16af22db8f6-kube-api-access-qcwqt\") pod \"multus-q574q\" (UID: \"f3999367-09f0-4979-8867-d16af22db8f6\") " pod="openshift-multus/multus-q574q" Jul 30 15:09:35.428597 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.428563 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:09:35.434363 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.434330 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 15:09:35.438650 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.438627 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-grsr9" Jul 30 15:09:35.447507 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.447477 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cvbfc" Jul 30 15:09:35.455017 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.454998 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g8lsb" Jul 30 15:09:35.462722 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.462700 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" Jul 30 15:09:35.469333 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.469306 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" Jul 30 15:09:35.477052 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.477023 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xj2rx" Jul 30 15:09:35.483655 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.483629 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qbrvz" Jul 30 15:09:35.488541 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.488522 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q574q" Jul 30 15:09:35.537995 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:35.537975 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693f4057_2498_479f_912c_c1f3f77aefab.slice/crio-aaf2efeb532350a6bb0783177dc7ee7fe3659c4649c817b94d08aab685df8ef9 WatchSource:0}: Error finding container aaf2efeb532350a6bb0783177dc7ee7fe3659c4649c817b94d08aab685df8ef9: Status 404 returned error can't find the container with id aaf2efeb532350a6bb0783177dc7ee7fe3659c4649c817b94d08aab685df8ef9 Jul 30 15:09:35.552425 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:35.552405 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadde37e6_1714_44ed_bb3d_57d3afcd1337.slice/crio-a16b4909cfd683f018a79d6eaf82e625e9517d3d6be26d85f2ee527a62701b34 WatchSource:0}: Error finding container a16b4909cfd683f018a79d6eaf82e625e9517d3d6be26d85f2ee527a62701b34: Status 404 returned error can't find the container with id a16b4909cfd683f018a79d6eaf82e625e9517d3d6be26d85f2ee527a62701b34 Jul 30 15:09:35.556716 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:35.556663 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd846133_3b4c_459a_abd2_58609374a7e8.slice/crio-45628936fcee4141872d60c2d9016be8949aa2e56d05d8c613a8c53107fff669 WatchSource:0}: Error finding container 45628936fcee4141872d60c2d9016be8949aa2e56d05d8c613a8c53107fff669: Status 404 returned error can't find the container with id 45628936fcee4141872d60c2d9016be8949aa2e56d05d8c613a8c53107fff669 Jul 30 15:09:35.557451 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:35.557428 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d7c93b_7592_4f23_aea2_96edecd05fe9.slice/crio-fc6489ebbd97e71cec1811ee701394694685ae5e93b1be07d666e76230f5ae57 WatchSource:0}: Error finding container fc6489ebbd97e71cec1811ee701394694685ae5e93b1be07d666e76230f5ae57: Status 404 returned error can't find the container with id fc6489ebbd97e71cec1811ee701394694685ae5e93b1be07d666e76230f5ae57 Jul 30 15:09:35.558295 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:09:35.558180 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb073d9a_53e8_4a4b_bc78_be972bc96d24.slice/crio-0b81de966641fa7f4ba6ef5d0ba97ae59ced7d949e39e7f2510c92b0786c9cc9 WatchSource:0}: Error finding container 0b81de966641fa7f4ba6ef5d0ba97ae59ced7d949e39e7f2510c92b0786c9cc9: Status 404 returned error can't find the container with id 0b81de966641fa7f4ba6ef5d0ba97ae59ced7d949e39e7f2510c92b0786c9cc9 Jul 30 15:09:35.746183 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.746155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs\") pod \"network-metrics-daemon-rsf9b\" (UID: \"a8cf4460-ac48-44e7-9a6e-64ff431ca403\") " pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:35.746333 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.746315 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 15:09:35.746402 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.746391 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs podName:a8cf4460-ac48-44e7-9a6e-64ff431ca403 nodeName:}" failed. No retries permitted until 2026-07-30 15:09:36.746372347 +0000 UTC m=+3.981503185 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs") pod "network-metrics-daemon-rsf9b" (UID: "a8cf4460-ac48-44e7-9a6e-64ff431ca403") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 15:09:35.847009 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.846847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxfc\" (UniqueName: \"kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc\") pod \"network-check-target-pc6xx\" (UID: \"496d6e44-f8b2-447b-b325-9b9be2c0e935\") " pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:35.847124 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.847012 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 15:09:35.847124 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.847036 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 15:09:35.847124 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.847048 2579 projected.go:196] Error preparing data for projected volume kube-api-access-xhxfc for pod openshift-network-diagnostics/network-check-target-pc6xx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 15:09:35.847124 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:35.847061 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret\") pod \"global-pull-secret-syncer-qh4bp\" (UID: \"113d506f-fc3f-4ea8-9eb2-98a534c8b5db\") " pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:35.847124 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.847111 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc podName:496d6e44-f8b2-447b-b325-9b9be2c0e935 nodeName:}" failed. No retries permitted until 2026-07-30 15:09:36.847094852 +0000 UTC m=+4.082225690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhxfc" (UniqueName: "kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc") pod "network-check-target-pc6xx" (UID: "496d6e44-f8b2-447b-b325-9b9be2c0e935") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 15:09:35.847364 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.847171 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 15:09:35.847364 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:35.847224 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret podName:113d506f-fc3f-4ea8-9eb2-98a534c8b5db nodeName:}" failed. No retries permitted until 2026-07-30 15:09:36.847212699 +0000 UTC m=+4.082343523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret") pod "global-pull-secret-syncer-qh4bp" (UID: "113d506f-fc3f-4ea8-9eb2-98a534c8b5db") : object "kube-system"/"original-pull-secret" not registered Jul 30 15:09:36.171906 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:36.171780 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 15:04:35 +0000 UTC" deadline="2028-03-05 02:37:05.892740731 +0000 UTC" Jul 30 15:09:36.171906 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:36.171867 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14003h27m29.720878513s" Jul 30 15:09:36.200588 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:36.200467 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" event={"ID":"adde37e6-1714-44ed-bb3d-57d3afcd1337","Type":"ContainerStarted","Data":"a16b4909cfd683f018a79d6eaf82e625e9517d3d6be26d85f2ee527a62701b34"} Jul 30 15:09:36.202435 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:36.202327 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" event={"ID":"693f4057-2498-479f-912c-c1f3f77aefab","Type":"ContainerStarted","Data":"aaf2efeb532350a6bb0783177dc7ee7fe3659c4649c817b94d08aab685df8ef9"} Jul 30 15:09:36.206311 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:36.205636 2579 generic.go:356] "Generic (PLEG): container finished" podID="cf1e64c012dc1860c2dcd3c41af1093c" containerID="15bb56f7bf39b7e86bd5cae4a77d6d78304611a82b772151a0fb435b501e3f62" exitCode=0 Jul 30 15:09:36.206311 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:36.205889 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal" event={"ID":"cf1e64c012dc1860c2dcd3c41af1093c","Type":"ContainerDied","Data":"15bb56f7bf39b7e86bd5cae4a77d6d78304611a82b772151a0fb435b501e3f62"} Jul 30 15:09:36.208505 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:36.208428 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbrvz" event={"ID":"e04c52a8-91f2-49a2-9e17-c015e0c44d36","Type":"ContainerStarted","Data":"24ebfd6d089900da96ea1a957bb3aa45e48be9036d16b1e4aa8b05ad77404892"} Jul 30 15:09:36.212084 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:36.212060 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" event={"ID":"01d7c93b-7592-4f23-aea2-96edecd05fe9","Type":"ContainerStarted","Data":"fc6489ebbd97e71cec1811ee701394694685ae5e93b1be07d666e76230f5ae57"} Jul 30 15:09:36.214279 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:36.214256 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g8lsb" event={"ID":"ec9478e9-d9e0-4ea7-9412-e36fc1a305c7","Type":"ContainerStarted","Data":"806f5ec0f1f17ec70b2dc14ee69efcbf73d9bcca86a1da30d0f396c113fd8f1c"} Jul 30 15:09:36.217216 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:36.217191 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q574q" event={"ID":"f3999367-09f0-4979-8867-d16af22db8f6","Type":"ContainerStarted","Data":"73d2a502744df7792c823af8641d32ef6d7385954f9313562386e82de12fec84"} Jul 30 15:09:36.220777 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:36.220753 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cvbfc" event={"ID":"c35ba36d-8199-471f-99df-70cc5ba0c05e","Type":"ContainerStarted","Data":"4e327d39b11e59051157ae6bf641b97d495c2b06a34450395ccc132aa39bedbb"} Jul 30 15:09:36.223619 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:36.223595 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-grsr9" event={"ID":"db073d9a-53e8-4a4b-bc78-be972bc96d24","Type":"ContainerStarted","Data":"0b81de966641fa7f4ba6ef5d0ba97ae59ced7d949e39e7f2510c92b0786c9cc9"} Jul 30 15:09:36.227974 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:36.227951 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xj2rx" event={"ID":"cd846133-3b4c-459a-abd2-58609374a7e8","Type":"ContainerStarted","Data":"45628936fcee4141872d60c2d9016be8949aa2e56d05d8c613a8c53107fff669"} Jul 30 15:09:36.755497 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:36.754829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs\") pod \"network-metrics-daemon-rsf9b\" (UID: \"a8cf4460-ac48-44e7-9a6e-64ff431ca403\") " pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:36.755497 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:36.755013 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 15:09:36.755497 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:36.755072 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs podName:a8cf4460-ac48-44e7-9a6e-64ff431ca403 nodeName:}" failed. No retries permitted until 2026-07-30 15:09:38.755054268 +0000 UTC m=+5.990185097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs") pod "network-metrics-daemon-rsf9b" (UID: "a8cf4460-ac48-44e7-9a6e-64ff431ca403") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 15:09:36.855910 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:36.855153 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret\") pod \"global-pull-secret-syncer-qh4bp\" (UID: \"113d506f-fc3f-4ea8-9eb2-98a534c8b5db\") " pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:36.855910 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:36.855231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxfc\" (UniqueName: \"kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc\") pod \"network-check-target-pc6xx\" (UID: \"496d6e44-f8b2-447b-b325-9b9be2c0e935\") " pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:36.855910 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:36.855375 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 15:09:36.855910 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:36.855395 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 15:09:36.855910 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:36.855408 2579 projected.go:196] Error preparing data for projected volume kube-api-access-xhxfc for pod openshift-network-diagnostics/network-check-target-pc6xx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 15:09:36.855910 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:36.855461 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc podName:496d6e44-f8b2-447b-b325-9b9be2c0e935 nodeName:}" failed. No retries permitted until 2026-07-30 15:09:38.855443253 +0000 UTC m=+6.090574084 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhxfc" (UniqueName: "kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc") pod "network-check-target-pc6xx" (UID: "496d6e44-f8b2-447b-b325-9b9be2c0e935") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 15:09:36.855910 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:36.855529 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 15:09:36.855910 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:36.855561 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret podName:113d506f-fc3f-4ea8-9eb2-98a534c8b5db nodeName:}" failed. No retries permitted until 2026-07-30 15:09:38.855550936 +0000 UTC m=+6.090681767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret") pod "global-pull-secret-syncer-qh4bp" (UID: "113d506f-fc3f-4ea8-9eb2-98a534c8b5db") : object "kube-system"/"original-pull-secret" not registered Jul 30 15:09:37.172039 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:37.171995 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 15:04:35 +0000 UTC" deadline="2028-04-09 05:52:10.517074564 +0000 UTC" Jul 30 15:09:37.172039 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:37.172038 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14846h42m33.34504076s" Jul 30 15:09:37.196000 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:37.195967 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:37.196147 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:37.196107 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsf9b" podUID="a8cf4460-ac48-44e7-9a6e-64ff431ca403" Jul 30 15:09:37.196421 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:37.195967 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:37.196421 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:37.196365 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4bp" podUID="113d506f-fc3f-4ea8-9eb2-98a534c8b5db" Jul 30 15:09:37.196421 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:37.196392 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:37.196611 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:37.196485 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc6xx" podUID="496d6e44-f8b2-447b-b325-9b9be2c0e935" Jul 30 15:09:37.246753 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:37.246103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal" event={"ID":"cf1e64c012dc1860c2dcd3c41af1093c","Type":"ContainerStarted","Data":"c7814f4a49c37da3c9615bf48003825fe00e610078146e7f4e1c1bcd211ab13c"} Jul 30 15:09:38.769838 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:38.769789 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs\") pod \"network-metrics-daemon-rsf9b\" (UID: \"a8cf4460-ac48-44e7-9a6e-64ff431ca403\") " pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:38.770284 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:38.770025 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 15:09:38.770284 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:38.770089 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs podName:a8cf4460-ac48-44e7-9a6e-64ff431ca403 nodeName:}" failed. No retries permitted until 2026-07-30 15:09:42.770070584 +0000 UTC m=+10.005201425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs") pod "network-metrics-daemon-rsf9b" (UID: "a8cf4460-ac48-44e7-9a6e-64ff431ca403") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 15:09:38.870554 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:38.870515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret\") pod \"global-pull-secret-syncer-qh4bp\" (UID: \"113d506f-fc3f-4ea8-9eb2-98a534c8b5db\") " pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:38.870752 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:38.870612 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxfc\" (UniqueName: \"kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc\") pod \"network-check-target-pc6xx\" (UID: \"496d6e44-f8b2-447b-b325-9b9be2c0e935\") " pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:38.870829 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:38.870766 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 15:09:38.870829 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:38.870787 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 15:09:38.870829 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:38.870814 2579 projected.go:196] Error preparing data for projected volume kube-api-access-xhxfc for pod openshift-network-diagnostics/network-check-target-pc6xx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 15:09:38.871151 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:38.870871 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc podName:496d6e44-f8b2-447b-b325-9b9be2c0e935 nodeName:}" failed. No retries permitted until 2026-07-30 15:09:42.870851946 +0000 UTC m=+10.105982769 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhxfc" (UniqueName: "kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc") pod "network-check-target-pc6xx" (UID: "496d6e44-f8b2-447b-b325-9b9be2c0e935") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 15:09:38.871151 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:38.870963 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 15:09:38.871151 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:38.871010 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret podName:113d506f-fc3f-4ea8-9eb2-98a534c8b5db nodeName:}" failed. No retries permitted until 2026-07-30 15:09:42.870995868 +0000 UTC m=+10.106126706 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret") pod "global-pull-secret-syncer-qh4bp" (UID: "113d506f-fc3f-4ea8-9eb2-98a534c8b5db") : object "kube-system"/"original-pull-secret" not registered Jul 30 15:09:39.193924 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:39.193111 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:39.193924 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:39.193239 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc6xx" podUID="496d6e44-f8b2-447b-b325-9b9be2c0e935" Jul 30 15:09:39.193924 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:39.193624 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:39.193924 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:39.193724 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsf9b" podUID="a8cf4460-ac48-44e7-9a6e-64ff431ca403" Jul 30 15:09:39.193924 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:39.193848 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:39.194344 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:39.193945 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4bp" podUID="113d506f-fc3f-4ea8-9eb2-98a534c8b5db" Jul 30 15:09:41.192037 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:41.192001 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:41.192573 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:41.192001 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:41.192573 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:41.192140 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:41.192573 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:41.192146 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsf9b" podUID="a8cf4460-ac48-44e7-9a6e-64ff431ca403" Jul 30 15:09:41.192573 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:41.192255 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4bp" podUID="113d506f-fc3f-4ea8-9eb2-98a534c8b5db" Jul 30 15:09:41.192573 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:41.192334 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc6xx" podUID="496d6e44-f8b2-447b-b325-9b9be2c0e935" Jul 30 15:09:42.798293 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:42.798252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs\") pod \"network-metrics-daemon-rsf9b\" (UID: \"a8cf4460-ac48-44e7-9a6e-64ff431ca403\") " pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:42.798740 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:42.798421 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 15:09:42.798740 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:42.798512 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs podName:a8cf4460-ac48-44e7-9a6e-64ff431ca403 nodeName:}" failed. No retries permitted until 2026-07-30 15:09:50.798491137 +0000 UTC m=+18.033621965 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs") pod "network-metrics-daemon-rsf9b" (UID: "a8cf4460-ac48-44e7-9a6e-64ff431ca403") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 15:09:42.899067 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:42.899026 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxfc\" (UniqueName: \"kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc\") pod \"network-check-target-pc6xx\" (UID: \"496d6e44-f8b2-447b-b325-9b9be2c0e935\") " pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:42.899255 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:42.899118 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret\") pod \"global-pull-secret-syncer-qh4bp\" (UID: \"113d506f-fc3f-4ea8-9eb2-98a534c8b5db\") " pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:42.899255 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:42.899211 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 15:09:42.899255 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:42.899235 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 15:09:42.899255 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:42.899240 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 15:09:42.899255 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:42.899249 2579 projected.go:196] Error preparing data for projected volume kube-api-access-xhxfc for pod openshift-network-diagnostics/network-check-target-pc6xx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 15:09:42.899504 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:42.899293 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret podName:113d506f-fc3f-4ea8-9eb2-98a534c8b5db nodeName:}" failed. No retries permitted until 2026-07-30 15:09:50.899275921 +0000 UTC m=+18.134406758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret") pod "global-pull-secret-syncer-qh4bp" (UID: "113d506f-fc3f-4ea8-9eb2-98a534c8b5db") : object "kube-system"/"original-pull-secret" not registered Jul 30 15:09:42.899504 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:42.899310 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc podName:496d6e44-f8b2-447b-b325-9b9be2c0e935 nodeName:}" failed. No retries permitted until 2026-07-30 15:09:50.8993022 +0000 UTC m=+18.134433034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhxfc" (UniqueName: "kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc") pod "network-check-target-pc6xx" (UID: "496d6e44-f8b2-447b-b325-9b9be2c0e935") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 15:09:43.192828 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:43.192766 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:43.193009 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:43.192907 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:43.193009 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:43.192932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:43.193137 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:43.192904 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsf9b" podUID="a8cf4460-ac48-44e7-9a6e-64ff431ca403" Jul 30 15:09:43.193137 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:43.193022 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4bp" podUID="113d506f-fc3f-4ea8-9eb2-98a534c8b5db" Jul 30 15:09:43.193137 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:43.193104 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc6xx" podUID="496d6e44-f8b2-447b-b325-9b9be2c0e935" Jul 30 15:09:45.191941 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:45.191910 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:45.192382 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:45.192047 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsf9b" podUID="a8cf4460-ac48-44e7-9a6e-64ff431ca403" Jul 30 15:09:45.192479 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:45.192438 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:45.192575 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:45.192530 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4bp" podUID="113d506f-fc3f-4ea8-9eb2-98a534c8b5db" Jul 30 15:09:45.192575 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:45.192565 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:45.192755 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:45.192671 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc6xx" podUID="496d6e44-f8b2-447b-b325-9b9be2c0e935" Jul 30 15:09:47.191782 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:47.191749 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:47.192256 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:47.191763 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:47.192256 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:47.191887 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:47.192256 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:47.191885 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc6xx" podUID="496d6e44-f8b2-447b-b325-9b9be2c0e935" Jul 30 15:09:47.192256 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:47.192004 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4bp" podUID="113d506f-fc3f-4ea8-9eb2-98a534c8b5db" Jul 30 15:09:47.192256 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:47.192101 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsf9b" podUID="a8cf4460-ac48-44e7-9a6e-64ff431ca403" Jul 30 15:09:49.192182 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:49.192142 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:49.192695 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:49.192197 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:49.192695 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:49.192280 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4bp" podUID="113d506f-fc3f-4ea8-9eb2-98a534c8b5db" Jul 30 15:09:49.192695 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:49.192416 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsf9b" podUID="a8cf4460-ac48-44e7-9a6e-64ff431ca403" Jul 30 15:09:49.192695 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:49.192458 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:49.192695 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:49.192522 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc6xx" podUID="496d6e44-f8b2-447b-b325-9b9be2c0e935" Jul 30 15:09:50.858061 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:50.858019 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs\") pod \"network-metrics-daemon-rsf9b\" (UID: \"a8cf4460-ac48-44e7-9a6e-64ff431ca403\") " pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:50.858501 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:50.858169 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 15:09:50.858501 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:50.858239 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs podName:a8cf4460-ac48-44e7-9a6e-64ff431ca403 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:06.858222517 +0000 UTC m=+34.093353340 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs") pod "network-metrics-daemon-rsf9b" (UID: "a8cf4460-ac48-44e7-9a6e-64ff431ca403") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 15:09:50.959127 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:50.959088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret\") pod \"global-pull-secret-syncer-qh4bp\" (UID: \"113d506f-fc3f-4ea8-9eb2-98a534c8b5db\") " pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:50.959289 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:50.959175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxfc\" (UniqueName: \"kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc\") pod \"network-check-target-pc6xx\" (UID: \"496d6e44-f8b2-447b-b325-9b9be2c0e935\") " pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:50.959289 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:50.959242 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 15:09:50.959414 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:50.959321 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret podName:113d506f-fc3f-4ea8-9eb2-98a534c8b5db nodeName:}" failed. No retries permitted until 2026-07-30 15:10:06.959301332 +0000 UTC m=+34.194432164 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret") pod "global-pull-secret-syncer-qh4bp" (UID: "113d506f-fc3f-4ea8-9eb2-98a534c8b5db") : object "kube-system"/"original-pull-secret" not registered Jul 30 15:09:50.959414 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:50.959322 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 15:09:50.959414 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:50.959347 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 15:09:50.959414 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:50.959361 2579 projected.go:196] Error preparing data for projected volume kube-api-access-xhxfc for pod openshift-network-diagnostics/network-check-target-pc6xx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 15:09:50.959414 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:50.959403 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc podName:496d6e44-f8b2-447b-b325-9b9be2c0e935 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:06.959391624 +0000 UTC m=+34.194522451 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhxfc" (UniqueName: "kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc") pod "network-check-target-pc6xx" (UID: "496d6e44-f8b2-447b-b325-9b9be2c0e935") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 15:09:51.192061 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:51.192016 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:51.192061 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:51.192025 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:51.192265 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:51.192150 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsf9b" podUID="a8cf4460-ac48-44e7-9a6e-64ff431ca403" Jul 30 15:09:51.192265 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:51.192206 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4bp" podUID="113d506f-fc3f-4ea8-9eb2-98a534c8b5db" Jul 30 15:09:51.192265 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:51.192251 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:51.192395 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:51.192339 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc6xx" podUID="496d6e44-f8b2-447b-b325-9b9be2c0e935" Jul 30 15:09:53.192384 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:53.192346 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:53.192841 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:53.192447 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc6xx" podUID="496d6e44-f8b2-447b-b325-9b9be2c0e935" Jul 30 15:09:53.192841 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:53.192468 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:53.192841 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:53.192544 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:53.192841 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:53.192654 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsf9b" podUID="a8cf4460-ac48-44e7-9a6e-64ff431ca403" Jul 30 15:09:53.192841 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:53.192743 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4bp" podUID="113d506f-fc3f-4ea8-9eb2-98a534c8b5db" Jul 30 15:09:55.191562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.191386 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:55.192134 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.191386 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:55.192134 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:55.191643 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc6xx" podUID="496d6e44-f8b2-447b-b325-9b9be2c0e935" Jul 30 15:09:55.192134 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.191466 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:55.192134 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:55.191751 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsf9b" podUID="a8cf4460-ac48-44e7-9a6e-64ff431ca403" Jul 30 15:09:55.192134 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:55.191845 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4bp" podUID="113d506f-fc3f-4ea8-9eb2-98a534c8b5db" Jul 30 15:09:55.279684 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.279657 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.ec2.internal" event={"ID":"27a35d60ff6bc9476dbc920db8c64fe2","Type":"ContainerStarted","Data":"7a242f06c9992bce12527fb314c90be0fbcf595956b0eaeccfd528b87c58f227"} Jul 30 15:09:55.281005 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.280984 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g8lsb" event={"ID":"ec9478e9-d9e0-4ea7-9412-e36fc1a305c7","Type":"ContainerStarted","Data":"f2e3f2a5e2876b7122e680d971292d278ce1798d073bbc32e8f5bfcdd258d687"} Jul 30 15:09:55.282190 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.282169 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q574q" event={"ID":"f3999367-09f0-4979-8867-d16af22db8f6","Type":"ContainerStarted","Data":"8f5612839663b9088eb6964d7c87938debd769216cae5d7e08b8b1457fdf20da"} Jul 30 15:09:55.283322 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.283303 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cvbfc" event={"ID":"c35ba36d-8199-471f-99df-70cc5ba0c05e","Type":"ContainerStarted","Data":"36f526e5ee57e6d3c18afdacd7795c225e799195b5676dade3893223b174ff8c"} Jul 30 15:09:55.284538 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.284519 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xj2rx" event={"ID":"cd846133-3b4c-459a-abd2-58609374a7e8","Type":"ContainerStarted","Data":"d8b04fdd64d4db8800fe6b24953681e927a99850ddf81ffcce5f6185d180cef2"} Jul 30 15:09:55.285786 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.285768 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" event={"ID":"adde37e6-1714-44ed-bb3d-57d3afcd1337","Type":"ContainerStarted","Data":"2dde7c00a489d6a6a872de3f96da21d1775e138ce26f16f05a4a00d88b657011"} Jul 30 15:09:55.286981 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.286961 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" event={"ID":"693f4057-2498-479f-912c-c1f3f77aefab","Type":"ContainerStarted","Data":"3171c2c16e4d48d9bd580f991d3a93c347e90943f1a7544b9fee8f97127180e1"} Jul 30 15:09:55.288433 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.288408 2579 generic.go:356] "Generic (PLEG): container finished" podID="e04c52a8-91f2-49a2-9e17-c015e0c44d36" containerID="a841be0fec196915a8122a3fddaf7cd373358d0ea15d7c564e13f069e0155229" exitCode=0 Jul 30 15:09:55.288528 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.288476 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbrvz" event={"ID":"e04c52a8-91f2-49a2-9e17-c015e0c44d36","Type":"ContainerDied","Data":"a841be0fec196915a8122a3fddaf7cd373358d0ea15d7c564e13f069e0155229"} Jul 30 15:09:55.289969 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.289952 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" event={"ID":"01d7c93b-7592-4f23-aea2-96edecd05fe9","Type":"ContainerStarted","Data":"81fa0f505b6323320f2fef599ca08114a6f2dee8a55758e605ed2d47a0dd154d"} Jul 30 15:09:55.293969 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.293934 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.ec2.internal" podStartSLOduration=21.293917607 podStartE2EDuration="21.293917607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 15:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:09:37.274436888 +0000 UTC m=+4.509567735" watchObservedRunningTime="2026-07-30 15:09:55.293917607 +0000 UTC m=+22.529048453" Jul 30 15:09:55.294207 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.294186 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.ec2.internal" podStartSLOduration=21.294179457 podStartE2EDuration="21.294179457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 15:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:09:55.293660189 +0000 UTC m=+22.528791036" watchObservedRunningTime="2026-07-30 15:09:55.294179457 +0000 UTC m=+22.529310303" Jul 30 15:09:55.307338 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.307307 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xj2rx" podStartSLOduration=3.645980525 podStartE2EDuration="22.307295566s" totalImagesPullingTime="18.661315041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:09:55.306858477 +0000 UTC m=+22.541989323" watchObservedRunningTime="2026-07-30 15:09:55.307295566 +0000 UTC m=+22.542426408" Jul 30 15:09:55.322483 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.322450 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-q574q" podStartSLOduration=3.449940158 podStartE2EDuration="22.322440857s" totalImagesPullingTime="18.872500699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 15:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:09:55.322222966 +0000 UTC m=+22.557353812" watchObservedRunningTime="2026-07-30 15:09:55.322440857 +0000 UTC m=+22.557571700" Jul 30 15:09:55.355653 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.355612 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cvbfc" podStartSLOduration=3.794149757 podStartE2EDuration="22.355594523s" totalImagesPullingTime="18.561444766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 15:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:09:55.340719286 +0000 UTC m=+22.575850134" watchObservedRunningTime="2026-07-30 15:09:55.355594523 +0000 UTC m=+22.590725372" Jul 30 15:09:55.373808 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.373763 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g8lsb" podStartSLOduration=11.602161856 podStartE2EDuration="22.373749428s" totalImagesPullingTime="10.771587572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 15:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:09:55.355087211 +0000 UTC m=+22.590218060" watchObservedRunningTime="2026-07-30 15:09:55.373749428 +0000 UTC m=+22.608880275" Jul 30 15:09:55.398071 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.398035 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zjqcf" podStartSLOduration=3.537513418 podStartE2EDuration="22.398021179s" totalImagesPullingTime="18.860507761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 15:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:09:55.373673425 +0000 UTC m=+22.608804272" watchObservedRunningTime="2026-07-30 15:09:55.398021179 +0000 UTC m=+22.633152024" Jul 30 15:09:55.783051 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:55.783027 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 30 15:09:56.186791 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:56.186682 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T15:09:55.783050111Z","UUID":"9f7db832-958e-4dee-bbc0-44d43adcc217","Handler":null,"Name":"","Endpoint":""} Jul 30 15:09:56.188548 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:56.188524 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 30 15:09:56.188685 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:56.188569 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 30 15:09:56.242855 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:56.242832 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xj2rx" Jul 30 15:09:56.243582 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:56.243541 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xj2rx" Jul 30 15:09:56.295814 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:56.295764 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" event={"ID":"01d7c93b-7592-4f23-aea2-96edecd05fe9","Type":"ContainerStarted","Data":"c2fe318da28fd0f7d82ad2f8d0bd05acde3cdaad2208a6c0b3a777126959d3f6"} Jul 30 15:09:56.296011 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:56.295864 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" event={"ID":"01d7c93b-7592-4f23-aea2-96edecd05fe9","Type":"ContainerStarted","Data":"d0677d69748b047fce4c5c7f321af0f252b09772dd2a99623bb15629cdc7db0c"} Jul 30 15:09:56.296011 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:56.295917 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" event={"ID":"01d7c93b-7592-4f23-aea2-96edecd05fe9","Type":"ContainerStarted","Data":"3dc9b9cfa13638e2b03f6ba85894da0b48465f62437862911b92d41abba47efe"} Jul 30 15:09:56.296011 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:56.295962 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" event={"ID":"01d7c93b-7592-4f23-aea2-96edecd05fe9","Type":"ContainerStarted","Data":"3f6ffae60a2e24c0405b6525e4a02c4736267c14df63c04855c67d7cb4c503c1"} Jul 30 15:09:56.296011 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:56.296004 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" event={"ID":"01d7c93b-7592-4f23-aea2-96edecd05fe9","Type":"ContainerStarted","Data":"f975979abdd743694fdc8ebab745934903287e7c11e09cac83ab8ad77cdefdde"} Jul 30 15:09:56.297311 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:56.297275 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-grsr9" event={"ID":"db073d9a-53e8-4a4b-bc78-be972bc96d24","Type":"ContainerStarted","Data":"3e0bc0e3eca3e255c4cb3a66e41e11065725c4168a7feab8991b87295d64bf8f"} Jul 30 15:09:56.299170 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:56.299148 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" event={"ID":"adde37e6-1714-44ed-bb3d-57d3afcd1337","Type":"ContainerStarted","Data":"ea654d0299ffd3b80526f68f8ac477555a01d8d0f8eb54906162768974bb6e64"} Jul 30 15:09:56.300054 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:56.300036 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xj2rx" Jul 30 15:09:56.300126 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:56.300086 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xj2rx" Jul 30 15:09:56.314144 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:56.314101 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-grsr9" podStartSLOduration=4.752538861 podStartE2EDuration="23.314084974s" totalImagesPullingTime="18.561546113s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 15:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:09:56.31319183 +0000 UTC m=+23.548322679" watchObservedRunningTime="2026-07-30 15:09:56.314084974 +0000 UTC m=+23.549215820" Jul 30 15:09:57.191338 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:57.191178 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:57.191609 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:57.191178 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:57.191609 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:57.191439 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsf9b" podUID="a8cf4460-ac48-44e7-9a6e-64ff431ca403" Jul 30 15:09:57.191609 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:57.191228 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:57.191609 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:57.191471 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4bp" podUID="113d506f-fc3f-4ea8-9eb2-98a534c8b5db" Jul 30 15:09:57.191609 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:57.191587 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc6xx" podUID="496d6e44-f8b2-447b-b325-9b9be2c0e935" Jul 30 15:09:57.303264 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:57.303230 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" event={"ID":"adde37e6-1714-44ed-bb3d-57d3afcd1337","Type":"ContainerStarted","Data":"975cab3669935c9ff7d3835870f43c8a3f1cb7e56b7c14e18bcd6e2eb2fc025d"} Jul 30 15:09:57.325068 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:57.325024 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57rvj" podStartSLOduration=3.68594372 podStartE2EDuration="24.325012112s" totalImagesPullingTime="20.639068392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 15:09:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:09:57.324712367 +0000 UTC m=+24.559843214" watchObservedRunningTime="2026-07-30 15:09:57.325012112 +0000 UTC m=+24.560142957" Jul 30 15:09:58.309953 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:58.309919 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" event={"ID":"01d7c93b-7592-4f23-aea2-96edecd05fe9","Type":"ContainerStarted","Data":"895d12b93d5a0defed750db773996102e3b0d3dff261c22aa8f46c5682694b76"} Jul 30 15:09:59.191489 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:59.191455 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:09:59.191489 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:59.191476 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:09:59.191726 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:09:59.191467 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:09:59.191726 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:59.191547 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc6xx" podUID="496d6e44-f8b2-447b-b325-9b9be2c0e935" Jul 30 15:09:59.191726 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:59.191652 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsf9b" podUID="a8cf4460-ac48-44e7-9a6e-64ff431ca403" Jul 30 15:09:59.191877 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:09:59.191742 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4bp" podUID="113d506f-fc3f-4ea8-9eb2-98a534c8b5db" Jul 30 15:10:00.317599 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:00.317574 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" event={"ID":"01d7c93b-7592-4f23-aea2-96edecd05fe9","Type":"ContainerStarted","Data":"f895e155d3761fdbbf45a42b7a240f73214dc86c712ce14465c84fea101752c5"} Jul 30 15:10:01.192066 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:01.191849 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:10:01.192278 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:01.191857 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:10:01.192278 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:01.192159 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc6xx" podUID="496d6e44-f8b2-447b-b325-9b9be2c0e935" Jul 30 15:10:01.192380 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:01.192273 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsf9b" podUID="a8cf4460-ac48-44e7-9a6e-64ff431ca403" Jul 30 15:10:01.192380 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:01.191930 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:10:01.192476 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:01.192400 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4bp" podUID="113d506f-fc3f-4ea8-9eb2-98a534c8b5db" Jul 30 15:10:01.320077 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:01.320039 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:10:01.320518 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:01.320124 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:10:01.338527 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:01.338497 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:10:01.353551 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:01.353509 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" podStartSLOduration=8.73686838 podStartE2EDuration="28.353496096s" totalImagesPullingTime="19.616627716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 15:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:10:01.352786604 +0000 UTC m=+28.587917451" watchObservedRunningTime="2026-07-30 15:10:01.353496096 +0000 UTC m=+28.588626941" Jul 30 15:10:02.322791 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:02.322762 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:10:02.327285 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:02.327261 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qh4bp"] Jul 30 15:10:02.327425 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:02.327370 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:10:02.327482 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:02.327464 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4bp" podUID="113d506f-fc3f-4ea8-9eb2-98a534c8b5db" Jul 30 15:10:02.329630 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:02.329388 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pc6xx"] Jul 30 15:10:02.329630 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:02.329497 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:10:02.329630 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:02.329587 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc6xx" podUID="496d6e44-f8b2-447b-b325-9b9be2c0e935" Jul 30 15:10:02.330288 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:02.330266 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rsf9b"] Jul 30 15:10:02.330385 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:02.330370 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:10:02.330738 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:02.330518 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsf9b" podUID="a8cf4460-ac48-44e7-9a6e-64ff431ca403" Jul 30 15:10:02.341982 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:02.341956 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:10:04.191952 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:04.191716 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:10:04.192312 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:04.191760 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:10:04.192312 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:04.191985 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4bp" podUID="113d506f-fc3f-4ea8-9eb2-98a534c8b5db" Jul 30 15:10:04.192312 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:04.191781 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:10:04.192312 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:04.192092 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc6xx" podUID="496d6e44-f8b2-447b-b325-9b9be2c0e935" Jul 30 15:10:04.192312 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:04.192183 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsf9b" podUID="a8cf4460-ac48-44e7-9a6e-64ff431ca403" Jul 30 15:10:04.328252 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:04.328222 2579 generic.go:356] "Generic (PLEG): container finished" podID="e04c52a8-91f2-49a2-9e17-c015e0c44d36" containerID="1ce234e3afbd8df83c0729adaa41ead96f1a8a41e0a95d3cae4be94fd2b1f34e" exitCode=0 Jul 30 15:10:04.328452 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:04.328309 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbrvz" event={"ID":"e04c52a8-91f2-49a2-9e17-c015e0c44d36","Type":"ContainerDied","Data":"1ce234e3afbd8df83c0729adaa41ead96f1a8a41e0a95d3cae4be94fd2b1f34e"} Jul 30 15:10:06.192006 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:06.191971 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:10:06.192006 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:06.192009 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:10:06.192595 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:06.192090 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4bp" podUID="113d506f-fc3f-4ea8-9eb2-98a534c8b5db" Jul 30 15:10:06.192595 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:06.192121 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc6xx" podUID="496d6e44-f8b2-447b-b325-9b9be2c0e935" Jul 30 15:10:06.192595 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:06.192158 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:10:06.192595 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:06.192245 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsf9b" podUID="a8cf4460-ac48-44e7-9a6e-64ff431ca403" Jul 30 15:10:06.335689 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:06.335645 2579 generic.go:356] "Generic (PLEG): container finished" podID="e04c52a8-91f2-49a2-9e17-c015e0c44d36" containerID="e36c34cf8dad7c3f17291d9bc9df5537c3aff7c0c2ece3633862795adf96cd75" exitCode=0 Jul 30 15:10:06.335888 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:06.335696 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbrvz" event={"ID":"e04c52a8-91f2-49a2-9e17-c015e0c44d36","Type":"ContainerDied","Data":"e36c34cf8dad7c3f17291d9bc9df5537c3aff7c0c2ece3633862795adf96cd75"} Jul 30 15:10:06.875016 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:06.874982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs\") pod \"network-metrics-daemon-rsf9b\" (UID: \"a8cf4460-ac48-44e7-9a6e-64ff431ca403\") " pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:10:06.875202 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:06.875151 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 15:10:06.875262 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:06.875219 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs podName:a8cf4460-ac48-44e7-9a6e-64ff431ca403 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:38.875199519 +0000 UTC m=+66.110330344 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs") pod "network-metrics-daemon-rsf9b" (UID: "a8cf4460-ac48-44e7-9a6e-64ff431ca403") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 15:10:06.976088 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:06.976059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret\") pod \"global-pull-secret-syncer-qh4bp\" (UID: \"113d506f-fc3f-4ea8-9eb2-98a534c8b5db\") " pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:10:06.976266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:06.976108 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxfc\" (UniqueName: \"kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc\") pod \"network-check-target-pc6xx\" (UID: \"496d6e44-f8b2-447b-b325-9b9be2c0e935\") " pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:10:06.976266 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:06.976207 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 15:10:06.976266 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:06.976209 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 15:10:06.976266 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:06.976222 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 15:10:06.976266 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:06.976233 2579 projected.go:196] Error preparing data for projected volume kube-api-access-xhxfc for pod openshift-network-diagnostics/network-check-target-pc6xx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 15:10:06.976442 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:06.976286 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret podName:113d506f-fc3f-4ea8-9eb2-98a534c8b5db nodeName:}" failed. No retries permitted until 2026-07-30 15:10:38.976263477 +0000 UTC m=+66.211394301 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret") pod "global-pull-secret-syncer-qh4bp" (UID: "113d506f-fc3f-4ea8-9eb2-98a534c8b5db") : object "kube-system"/"original-pull-secret" not registered Jul 30 15:10:06.976442 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:06.976306 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc podName:496d6e44-f8b2-447b-b325-9b9be2c0e935 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:38.976296931 +0000 UTC m=+66.211427757 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhxfc" (UniqueName: "kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc") pod "network-check-target-pc6xx" (UID: "496d6e44-f8b2-447b-b325-9b9be2c0e935") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 15:10:07.339717 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:07.339626 2579 generic.go:356] "Generic (PLEG): container finished" podID="e04c52a8-91f2-49a2-9e17-c015e0c44d36" containerID="292e0ca45fb91a4a3e9f7c8bae5b6f835540b93893a8bafc8032e9197b378965" exitCode=0 Jul 30 15:10:07.339717 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:07.339696 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbrvz" event={"ID":"e04c52a8-91f2-49a2-9e17-c015e0c44d36","Type":"ContainerDied","Data":"292e0ca45fb91a4a3e9f7c8bae5b6f835540b93893a8bafc8032e9197b378965"} Jul 30 15:10:08.084251 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.084225 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-244.ec2.internal" event="NodeReady" Jul 30 15:10:08.084389 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.084360 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 30 15:10:08.122055 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.122023 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f875d6977-b47mk"] Jul 30 15:10:08.124178 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.124160 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.126941 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.126919 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 30 15:10:08.127120 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.127102 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 30 15:10:08.127382 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.127344 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9t9xr\"" Jul 30 15:10:08.127530 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.127511 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 30 15:10:08.136418 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.136399 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gq5ph"] Jul 30 15:10:08.137240 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.137220 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 30 15:10:08.138256 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.138242 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ksbjt"] Jul 30 15:10:08.138358 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.138344 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gq5ph" Jul 30 15:10:08.140049 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.140030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ksbjt" Jul 30 15:10:08.140916 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.140898 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f875d6977-b47mk"] Jul 30 15:10:08.141435 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.141416 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 30 15:10:08.141524 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.141495 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 30 15:10:08.141607 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.141589 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7hzbg\"" Jul 30 15:10:08.142877 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.142858 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 30 15:10:08.143285 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.142895 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hpfnt\"" Jul 30 15:10:08.143285 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.142899 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 30 15:10:08.143285 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.143270 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 30 15:10:08.147560 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.147540 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gq5ph"] Jul 30 15:10:08.155936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.155916 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ksbjt"] Jul 30 15:10:08.191420 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.191401 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:10:08.191548 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.191402 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:10:08.191625 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.191408 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:10:08.194351 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.194334 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 15:10:08.194560 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.194539 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 15:10:08.194560 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.194550 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 15:10:08.194709 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.194598 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kvhp8\"" Jul 30 15:10:08.194709 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.194691 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-52dkp\"" Jul 30 15:10:08.194827 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.194728 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 15:10:08.284395 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.284326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwlmw\" (UniqueName: \"kubernetes.io/projected/d5098562-bcb2-4825-8429-184d57da5659-kube-api-access-qwlmw\") pod \"ingress-canary-ksbjt\" (UID: \"d5098562-bcb2-4825-8429-184d57da5659\") " pod="openshift-ingress-canary/ingress-canary-ksbjt" Jul 30 15:10:08.284395 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.284362 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:10:08.284570 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.284431 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.284570 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.284455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829b9cdd-657c-4f62-8a83-813c43eefa09-tmp-dir\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:10:08.284570 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.284477 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/406028a3-cf3c-44ee-83da-39aae2661e87-trusted-ca\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.284570 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.284499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/406028a3-cf3c-44ee-83da-39aae2661e87-ca-trust-extracted\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.284570 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.284525 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/406028a3-cf3c-44ee-83da-39aae2661e87-installation-pull-secrets\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.284758 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.284572 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-bound-sa-token\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.284758 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.284639 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckch4\" (UniqueName: \"kubernetes.io/projected/829b9cdd-657c-4f62-8a83-813c43eefa09-kube-api-access-ckch4\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:10:08.284758 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.284667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlbmv\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-kube-api-access-mlbmv\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.284758 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.284707 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/406028a3-cf3c-44ee-83da-39aae2661e87-registry-certificates\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.284932 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.284769 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5098562-bcb2-4825-8429-184d57da5659-cert\") pod \"ingress-canary-ksbjt\" (UID: \"d5098562-bcb2-4825-8429-184d57da5659\") " pod="openshift-ingress-canary/ingress-canary-ksbjt" Jul 30 15:10:08.284932 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.284813 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/829b9cdd-657c-4f62-8a83-813c43eefa09-config-volume\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:10:08.284932 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.284853 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/406028a3-cf3c-44ee-83da-39aae2661e87-image-registry-private-configuration\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.385832 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.385786 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwlmw\" (UniqueName: \"kubernetes.io/projected/d5098562-bcb2-4825-8429-184d57da5659-kube-api-access-qwlmw\") pod \"ingress-canary-ksbjt\" (UID: \"d5098562-bcb2-4825-8429-184d57da5659\") " pod="openshift-ingress-canary/ingress-canary-ksbjt" Jul 30 15:10:08.386478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.385853 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:10:08.386478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.385900 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.386478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.385925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829b9cdd-657c-4f62-8a83-813c43eefa09-tmp-dir\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:10:08.386478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.386010 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/406028a3-cf3c-44ee-83da-39aae2661e87-trusted-ca\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.386478 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:08.386029 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 15:10:08.386478 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:08.386045 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f875d6977-b47mk: secret "image-registry-tls" not found Jul 30 15:10:08.386478 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:08.386110 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls podName:406028a3-cf3c-44ee-83da-39aae2661e87 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:08.8860922 +0000 UTC m=+36.121223027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls") pod "image-registry-f875d6977-b47mk" (UID: "406028a3-cf3c-44ee-83da-39aae2661e87") : secret "image-registry-tls" not found Jul 30 15:10:08.386478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.386128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/406028a3-cf3c-44ee-83da-39aae2661e87-ca-trust-extracted\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.386478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.386161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/406028a3-cf3c-44ee-83da-39aae2661e87-installation-pull-secrets\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.386478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.386186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-bound-sa-token\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.386478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.386222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckch4\" (UniqueName: \"kubernetes.io/projected/829b9cdd-657c-4f62-8a83-813c43eefa09-kube-api-access-ckch4\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:10:08.386478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.386248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlbmv\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-kube-api-access-mlbmv\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.386478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.386276 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/406028a3-cf3c-44ee-83da-39aae2661e87-registry-certificates\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.386478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.386329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5098562-bcb2-4825-8429-184d57da5659-cert\") pod \"ingress-canary-ksbjt\" (UID: \"d5098562-bcb2-4825-8429-184d57da5659\") " pod="openshift-ingress-canary/ingress-canary-ksbjt" Jul 30 15:10:08.386478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.386356 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/829b9cdd-657c-4f62-8a83-813c43eefa09-config-volume\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:10:08.386478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.386380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/406028a3-cf3c-44ee-83da-39aae2661e87-image-registry-private-configuration\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.386478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.386404 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829b9cdd-657c-4f62-8a83-813c43eefa09-tmp-dir\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:10:08.387340 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.386708 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/406028a3-cf3c-44ee-83da-39aae2661e87-ca-trust-extracted\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.387340 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.386953 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/406028a3-cf3c-44ee-83da-39aae2661e87-registry-certificates\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.387340 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.387009 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/406028a3-cf3c-44ee-83da-39aae2661e87-trusted-ca\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.387767 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.387602 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/829b9cdd-657c-4f62-8a83-813c43eefa09-config-volume\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:10:08.387767 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:08.387686 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 15:10:08.387767 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:08.387737 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls podName:829b9cdd-657c-4f62-8a83-813c43eefa09 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:08.887721855 +0000 UTC m=+36.122852684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls") pod "dns-default-gq5ph" (UID: "829b9cdd-657c-4f62-8a83-813c43eefa09") : secret "dns-default-metrics-tls" not found Jul 30 15:10:08.391049 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.391011 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/406028a3-cf3c-44ee-83da-39aae2661e87-image-registry-private-configuration\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.391049 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.391037 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5098562-bcb2-4825-8429-184d57da5659-cert\") pod \"ingress-canary-ksbjt\" (UID: \"d5098562-bcb2-4825-8429-184d57da5659\") " pod="openshift-ingress-canary/ingress-canary-ksbjt" Jul 30 15:10:08.391290 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.391040 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/406028a3-cf3c-44ee-83da-39aae2661e87-installation-pull-secrets\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.394692 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.394667 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwlmw\" (UniqueName: \"kubernetes.io/projected/d5098562-bcb2-4825-8429-184d57da5659-kube-api-access-qwlmw\") pod \"ingress-canary-ksbjt\" (UID: \"d5098562-bcb2-4825-8429-184d57da5659\") " pod="openshift-ingress-canary/ingress-canary-ksbjt" Jul 30 15:10:08.395724 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.395700 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-bound-sa-token\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.396296 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.396275 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckch4\" (UniqueName: \"kubernetes.io/projected/829b9cdd-657c-4f62-8a83-813c43eefa09-kube-api-access-ckch4\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:10:08.396461 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.396441 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlbmv\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-kube-api-access-mlbmv\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.453640 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.453609 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ksbjt" Jul 30 15:10:08.602490 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.602459 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ksbjt"] Jul 30 15:10:08.605828 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:10:08.605787 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5098562_bcb2_4825_8429_184d57da5659.slice/crio-ab3a529c620632c9ff2c80433136442aad2cd2d42ff6f3a94fe742af6a656fc4 WatchSource:0}: Error finding container ab3a529c620632c9ff2c80433136442aad2cd2d42ff6f3a94fe742af6a656fc4: Status 404 returned error can't find the container with id ab3a529c620632c9ff2c80433136442aad2cd2d42ff6f3a94fe742af6a656fc4 Jul 30 15:10:08.889994 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.889959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:10:08.890173 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:08.890012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:08.890173 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:08.890127 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 15:10:08.890298 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:08.890177 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 15:10:08.890298 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:08.890198 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f875d6977-b47mk: secret "image-registry-tls" not found Jul 30 15:10:08.890298 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:08.890201 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls podName:829b9cdd-657c-4f62-8a83-813c43eefa09 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:09.890182126 +0000 UTC m=+37.125312962 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls") pod "dns-default-gq5ph" (UID: "829b9cdd-657c-4f62-8a83-813c43eefa09") : secret "dns-default-metrics-tls" not found Jul 30 15:10:08.890298 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:08.890252 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls podName:406028a3-cf3c-44ee-83da-39aae2661e87 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:09.890232764 +0000 UTC m=+37.125363606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls") pod "image-registry-f875d6977-b47mk" (UID: "406028a3-cf3c-44ee-83da-39aae2661e87") : secret "image-registry-tls" not found Jul 30 15:10:09.346049 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:09.345959 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ksbjt" event={"ID":"d5098562-bcb2-4825-8429-184d57da5659","Type":"ContainerStarted","Data":"ab3a529c620632c9ff2c80433136442aad2cd2d42ff6f3a94fe742af6a656fc4"} Jul 30 15:10:09.898001 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:09.897961 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:10:09.898466 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:09.898027 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:09.898466 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:09.898132 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 15:10:09.898466 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:09.898146 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 15:10:09.898466 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:09.898160 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f875d6977-b47mk: secret "image-registry-tls" not found Jul 30 15:10:09.898466 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:09.898203 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls podName:829b9cdd-657c-4f62-8a83-813c43eefa09 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:11.898182064 +0000 UTC m=+39.133312895 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls") pod "dns-default-gq5ph" (UID: "829b9cdd-657c-4f62-8a83-813c43eefa09") : secret "dns-default-metrics-tls" not found Jul 30 15:10:09.898466 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:09.898223 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls podName:406028a3-cf3c-44ee-83da-39aae2661e87 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:11.898213404 +0000 UTC m=+39.133344232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls") pod "image-registry-f875d6977-b47mk" (UID: "406028a3-cf3c-44ee-83da-39aae2661e87") : secret "image-registry-tls" not found Jul 30 15:10:11.915184 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:11.915141 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:10:11.915603 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:11.915206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:11.915603 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:11.915320 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 15:10:11.915603 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:11.915347 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 15:10:11.915603 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:11.915361 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f875d6977-b47mk: secret "image-registry-tls" not found Jul 30 15:10:11.915603 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:11.915405 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls podName:829b9cdd-657c-4f62-8a83-813c43eefa09 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:15.915381033 +0000 UTC m=+43.150511863 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls") pod "dns-default-gq5ph" (UID: "829b9cdd-657c-4f62-8a83-813c43eefa09") : secret "dns-default-metrics-tls" not found Jul 30 15:10:11.915603 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:11.915426 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls podName:406028a3-cf3c-44ee-83da-39aae2661e87 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:15.915416318 +0000 UTC m=+43.150547147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls") pod "image-registry-f875d6977-b47mk" (UID: "406028a3-cf3c-44ee-83da-39aae2661e87") : secret "image-registry-tls" not found Jul 30 15:10:13.228412 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:13.228379 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xj2rx_cd846133-3b4c-459a-abd2-58609374a7e8/konnectivity-agent/0.log" limit=1024 Jul 30 15:10:14.362193 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:14.362160 2579 generic.go:356] "Generic (PLEG): container finished" podID="e04c52a8-91f2-49a2-9e17-c015e0c44d36" containerID="4c026bdc1c799477ca499cf817d9050cb5fb78e3f38310a94d6a8e733c23f69b" exitCode=0 Jul 30 15:10:14.362645 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:14.362244 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbrvz" event={"ID":"e04c52a8-91f2-49a2-9e17-c015e0c44d36","Type":"ContainerDied","Data":"4c026bdc1c799477ca499cf817d9050cb5fb78e3f38310a94d6a8e733c23f69b"} Jul 30 15:10:14.363704 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:14.363685 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ksbjt" event={"ID":"d5098562-bcb2-4825-8429-184d57da5659","Type":"ContainerStarted","Data":"fd76d2235fa3a9c99751a0bf1cc843313ad84c637876519433b159dfbf494486"} Jul 30 15:10:14.399113 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:14.399059 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ksbjt" podStartSLOduration=1.703659 podStartE2EDuration="6.399044876s" totalImagesPullingTime="4.695385876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:10:14.398516606 +0000 UTC m=+41.633647453" watchObservedRunningTime="2026-07-30 15:10:14.399044876 +0000 UTC m=+41.634175722" Jul 30 15:10:14.504429 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:14.504392 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xj2rx_cd846133-3b4c-459a-abd2-58609374a7e8/konnectivity-agent/0.log" limit=1024 Jul 30 15:10:15.369080 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:15.369045 2579 generic.go:356] "Generic (PLEG): container finished" podID="e04c52a8-91f2-49a2-9e17-c015e0c44d36" containerID="1f22971d95ac254cd5a9c85b22aa5934970c25ef6fdb254c7e7fee061b1dee5e" exitCode=0 Jul 30 15:10:15.369492 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:15.369133 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbrvz" event={"ID":"e04c52a8-91f2-49a2-9e17-c015e0c44d36","Type":"ContainerDied","Data":"1f22971d95ac254cd5a9c85b22aa5934970c25ef6fdb254c7e7fee061b1dee5e"} Jul 30 15:10:15.945522 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:15.945489 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:10:15.945668 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:15.945531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:15.945668 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:15.945641 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 15:10:15.945738 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:15.945677 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 15:10:15.945738 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:15.945687 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f875d6977-b47mk: secret "image-registry-tls" not found Jul 30 15:10:15.945738 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:15.945702 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls podName:829b9cdd-657c-4f62-8a83-813c43eefa09 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:23.945687685 +0000 UTC m=+51.180818510 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls") pod "dns-default-gq5ph" (UID: "829b9cdd-657c-4f62-8a83-813c43eefa09") : secret "dns-default-metrics-tls" not found Jul 30 15:10:15.945738 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:15.945723 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls podName:406028a3-cf3c-44ee-83da-39aae2661e87 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:23.945712753 +0000 UTC m=+51.180843578 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls") pod "image-registry-f875d6977-b47mk" (UID: "406028a3-cf3c-44ee-83da-39aae2661e87") : secret "image-registry-tls" not found Jul 30 15:10:16.374743 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:16.374713 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbrvz" event={"ID":"e04c52a8-91f2-49a2-9e17-c015e0c44d36","Type":"ContainerStarted","Data":"51bcc50bd55f55eca19a2e21502db9705da8fed7319b5c9f4649c419df8f373c"} Jul 30 15:10:16.399679 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:16.399638 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qbrvz" podStartSLOduration=8.420310287 podStartE2EDuration="43.399624877s" totalImagesPullingTime="34.97931459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 15:09:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:10:16.397509714 +0000 UTC m=+43.632640572" watchObservedRunningTime="2026-07-30 15:10:16.399624877 +0000 UTC m=+43.634755722" Jul 30 15:10:16.526119 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:16.526089 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xj2rx_cd846133-3b4c-459a-abd2-58609374a7e8/konnectivity-agent/0.log" limit=1024 Jul 30 15:10:16.625696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:16.625612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xj2rx_cd846133-3b4c-459a-abd2-58609374a7e8/konnectivity-agent/0.log" limit=1024 Jul 30 15:10:18.695161 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:18.695118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xj2rx_cd846133-3b4c-459a-abd2-58609374a7e8/konnectivity-agent/0.log" limit=1024 Jul 30 15:10:18.767359 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:18.767328 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xj2rx_cd846133-3b4c-459a-abd2-58609374a7e8/konnectivity-agent/0.log" limit=1024 Jul 30 15:10:18.854683 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:18.854648 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xj2rx_cd846133-3b4c-459a-abd2-58609374a7e8/konnectivity-agent/0.log" limit=1024 Jul 30 15:10:18.933544 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:18.933517 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xj2rx_cd846133-3b4c-459a-abd2-58609374a7e8/konnectivity-agent/0.log" limit=1024 Jul 30 15:10:19.034575 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:19.034498 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xj2rx_cd846133-3b4c-459a-abd2-58609374a7e8/konnectivity-agent/0.log" limit=1024 Jul 30 15:10:19.135342 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:19.135309 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xj2rx_cd846133-3b4c-459a-abd2-58609374a7e8/konnectivity-agent/0.log" limit=1024 Jul 30 15:10:19.668925 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:19.668890 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xj2rx_cd846133-3b4c-459a-abd2-58609374a7e8/konnectivity-agent/0.log" limit=1024 Jul 30 15:10:19.733338 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:19.733313 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xj2rx_cd846133-3b4c-459a-abd2-58609374a7e8/konnectivity-agent/0.log" limit=1024 Jul 30 15:10:20.067651 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:20.067579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xj2rx_cd846133-3b4c-459a-abd2-58609374a7e8/konnectivity-agent/0.log" limit=1024 Jul 30 15:10:20.619960 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:20.619930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xj2rx_cd846133-3b4c-459a-abd2-58609374a7e8/konnectivity-agent/0.log" limit=1024 Jul 30 15:10:21.890787 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:21.890753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xj2rx_cd846133-3b4c-459a-abd2-58609374a7e8/konnectivity-agent/0.log" limit=1024 Jul 30 15:10:21.991740 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:21.991710 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xj2rx_cd846133-3b4c-459a-abd2-58609374a7e8/konnectivity-agent/0.log" limit=1024 Jul 30 15:10:23.999361 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:23.999324 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:10:23.999836 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:23.999370 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:23.999836 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:23.999480 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 15:10:23.999836 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:23.999481 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 15:10:23.999836 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:23.999552 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls podName:829b9cdd-657c-4f62-8a83-813c43eefa09 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:39.999536947 +0000 UTC m=+67.234667771 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls") pod "dns-default-gq5ph" (UID: "829b9cdd-657c-4f62-8a83-813c43eefa09") : secret "dns-default-metrics-tls" not found Jul 30 15:10:23.999836 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:23.999490 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f875d6977-b47mk: secret "image-registry-tls" not found Jul 30 15:10:23.999836 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:23.999620 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls podName:406028a3-cf3c-44ee-83da-39aae2661e87 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:39.99960793 +0000 UTC m=+67.234738753 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls") pod "image-registry-f875d6977-b47mk" (UID: "406028a3-cf3c-44ee-83da-39aae2661e87") : secret "image-registry-tls" not found Jul 30 15:10:34.339036 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:34.339007 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pbpn" Jul 30 15:10:38.902533 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:38.902498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs\") pod \"network-metrics-daemon-rsf9b\" (UID: \"a8cf4460-ac48-44e7-9a6e-64ff431ca403\") " pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:10:38.905538 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:38.905520 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 15:10:38.912988 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:38.912969 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 30 15:10:38.913060 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:38.913028 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs podName:a8cf4460-ac48-44e7-9a6e-64ff431ca403 nodeName:}" failed. No retries permitted until 2026-07-30 15:11:42.913014314 +0000 UTC m=+130.148145139 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs") pod "network-metrics-daemon-rsf9b" (UID: "a8cf4460-ac48-44e7-9a6e-64ff431ca403") : secret "metrics-daemon-secret" not found Jul 30 15:10:39.002951 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:39.002916 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret\") pod \"global-pull-secret-syncer-qh4bp\" (UID: \"113d506f-fc3f-4ea8-9eb2-98a534c8b5db\") " pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:10:39.003115 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:39.002976 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxfc\" (UniqueName: \"kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc\") pod \"network-check-target-pc6xx\" (UID: \"496d6e44-f8b2-447b-b325-9b9be2c0e935\") " pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:10:39.006046 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:39.006027 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 15:10:39.006105 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:39.006028 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 15:10:39.015659 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:39.015636 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 15:10:39.016334 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:39.016315 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/113d506f-fc3f-4ea8-9eb2-98a534c8b5db-original-pull-secret\") pod \"global-pull-secret-syncer-qh4bp\" (UID: \"113d506f-fc3f-4ea8-9eb2-98a534c8b5db\") " pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:10:39.025748 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:39.025727 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhxfc\" (UniqueName: \"kubernetes.io/projected/496d6e44-f8b2-447b-b325-9b9be2c0e935-kube-api-access-xhxfc\") pod \"network-check-target-pc6xx\" (UID: \"496d6e44-f8b2-447b-b325-9b9be2c0e935\") " pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:10:39.106470 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:39.103984 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-52dkp\"" Jul 30 15:10:39.110373 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:39.110357 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4bp" Jul 30 15:10:39.110886 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:39.110869 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:10:39.239594 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:39.239547 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pc6xx"] Jul 30 15:10:39.242433 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:10:39.242407 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496d6e44_f8b2_447b_b325_9b9be2c0e935.slice/crio-aa57cab578854dff99fbb994906a1e409f99e593e9eed0a9c6978931de97db54 WatchSource:0}: Error finding container aa57cab578854dff99fbb994906a1e409f99e593e9eed0a9c6978931de97db54: Status 404 returned error can't find the container with id aa57cab578854dff99fbb994906a1e409f99e593e9eed0a9c6978931de97db54 Jul 30 15:10:39.252425 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:39.251705 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qh4bp"] Jul 30 15:10:39.255704 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:10:39.255682 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113d506f_fc3f_4ea8_9eb2_98a534c8b5db.slice/crio-1f265d9a67898f364741e8803418d2cb49da33b67277ec11b8d52c60939c7d32 WatchSource:0}: Error finding container 1f265d9a67898f364741e8803418d2cb49da33b67277ec11b8d52c60939c7d32: Status 404 returned error can't find the container with id 1f265d9a67898f364741e8803418d2cb49da33b67277ec11b8d52c60939c7d32 Jul 30 15:10:39.426505 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:39.426439 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pc6xx" event={"ID":"496d6e44-f8b2-447b-b325-9b9be2c0e935","Type":"ContainerStarted","Data":"aa57cab578854dff99fbb994906a1e409f99e593e9eed0a9c6978931de97db54"} Jul 30 15:10:39.427464 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:39.427443 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qh4bp" event={"ID":"113d506f-fc3f-4ea8-9eb2-98a534c8b5db","Type":"ContainerStarted","Data":"1f265d9a67898f364741e8803418d2cb49da33b67277ec11b8d52c60939c7d32"} Jul 30 15:10:40.009867 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:40.009825 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:10:40.010392 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:40.009886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:10:40.010392 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:40.009969 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 15:10:40.010392 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:40.010000 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 15:10:40.010392 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:40.010013 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f875d6977-b47mk: secret "image-registry-tls" not found Jul 30 15:10:40.010392 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:40.010034 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls podName:829b9cdd-657c-4f62-8a83-813c43eefa09 nodeName:}" failed. No retries permitted until 2026-07-30 15:11:12.010017899 +0000 UTC m=+99.245148723 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls") pod "dns-default-gq5ph" (UID: "829b9cdd-657c-4f62-8a83-813c43eefa09") : secret "dns-default-metrics-tls" not found Jul 30 15:10:40.010392 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:40.010051 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls podName:406028a3-cf3c-44ee-83da-39aae2661e87 nodeName:}" failed. No retries permitted until 2026-07-30 15:11:12.010040245 +0000 UTC m=+99.245171072 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls") pod "image-registry-f875d6977-b47mk" (UID: "406028a3-cf3c-44ee-83da-39aae2661e87") : secret "image-registry-tls" not found Jul 30 15:10:44.408989 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.408958 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5c8b769f9f-tfnst"] Jul 30 15:10:44.411990 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.411973 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:44.415468 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.415444 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 30 15:10:44.415468 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.415466 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 30 15:10:44.415806 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.415781 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 30 15:10:44.416079 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.416063 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 30 15:10:44.416496 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.416480 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 30 15:10:44.416849 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.416834 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xtsqs\"" Jul 30 15:10:44.416903 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.416872 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 30 15:10:44.423829 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.423792 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c8b769f9f-tfnst"] Jul 30 15:10:44.441463 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.441432 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pc6xx" event={"ID":"496d6e44-f8b2-447b-b325-9b9be2c0e935","Type":"ContainerStarted","Data":"b605bbfa34a270fc9a7c6520fe6ce90012fa81b5a24ffac7b1f7a820078e69e6"} Jul 30 15:10:44.441569 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.441483 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:10:44.442821 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.442788 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qh4bp" event={"ID":"113d506f-fc3f-4ea8-9eb2-98a534c8b5db","Type":"ContainerStarted","Data":"2399974c06f4f2b945c4b8fa9870d25ca4b08393853de331225b57bc00ed8fa2"} Jul 30 15:10:44.447988 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.447961 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-stats-auth\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:44.448071 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.448029 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpw27\" (UniqueName: \"kubernetes.io/projected/04216c4f-4a2e-4c3b-a254-b2c11ad44244-kube-api-access-xpw27\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:44.448071 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.448054 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-default-certificate\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:44.448144 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.448073 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:44.448144 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.448090 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:44.471368 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.471326 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pc6xx" podStartSLOduration=66.69821607 podStartE2EDuration="1m11.47131499s" totalImagesPullingTime="4.77309892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:10:44.47086513 +0000 UTC m=+71.705995976" watchObservedRunningTime="2026-07-30 15:10:44.47131499 +0000 UTC m=+71.706445833" Jul 30 15:10:44.518728 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.518680 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qh4bp" podStartSLOduration=66.759505975 podStartE2EDuration="1m11.518665835s" totalImagesPullingTime="4.75915986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 15:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:10:44.518504589 +0000 UTC m=+71.753635434" watchObservedRunningTime="2026-07-30 15:10:44.518665835 +0000 UTC m=+71.753796680" Jul 30 15:10:44.548924 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.548891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-default-certificate\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:44.549030 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.548933 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:44.549069 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:44.549027 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 15:10:44.549069 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.549052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:44.549139 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:44.549078 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs podName:04216c4f-4a2e-4c3b-a254-b2c11ad44244 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:45.049063127 +0000 UTC m=+72.284193951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs") pod "router-default-5c8b769f9f-tfnst" (UID: "04216c4f-4a2e-4c3b-a254-b2c11ad44244") : secret "router-metrics-certs-default" not found Jul 30 15:10:44.549139 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.549132 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-stats-auth\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:44.549244 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:44.549156 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle podName:04216c4f-4a2e-4c3b-a254-b2c11ad44244 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:45.049146029 +0000 UTC m=+72.284276853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle") pod "router-default-5c8b769f9f-tfnst" (UID: "04216c4f-4a2e-4c3b-a254-b2c11ad44244") : configmap references non-existent config key: service-ca.crt Jul 30 15:10:44.549300 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.549252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpw27\" (UniqueName: \"kubernetes.io/projected/04216c4f-4a2e-4c3b-a254-b2c11ad44244-kube-api-access-xpw27\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:44.551284 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.551259 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-default-certificate\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:44.551393 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.551357 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-stats-auth\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:44.571446 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.571426 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpw27\" (UniqueName: \"kubernetes.io/projected/04216c4f-4a2e-4c3b-a254-b2c11ad44244-kube-api-access-xpw27\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:44.615213 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.615180 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q"] Jul 30 15:10:44.638420 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.638395 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q"] Jul 30 15:10:44.638530 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.638486 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q" Jul 30 15:10:44.646088 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.646066 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-txl6p\"" Jul 30 15:10:44.646201 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.646096 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 30 15:10:44.646363 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.646349 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 30 15:10:44.646959 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.646942 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 30 15:10:44.647019 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.646965 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 30 15:10:44.750601 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.750518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cc61754-0414-4c86-84de-250ba68dc560-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q\" (UID: \"0cc61754-0414-4c86-84de-250ba68dc560\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q" Jul 30 15:10:44.750601 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.750557 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0cc61754-0414-4c86-84de-250ba68dc560-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q\" (UID: \"0cc61754-0414-4c86-84de-250ba68dc560\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q" Jul 30 15:10:44.750788 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.750660 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjxlj\" (UniqueName: \"kubernetes.io/projected/0cc61754-0414-4c86-84de-250ba68dc560-kube-api-access-sjxlj\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q\" (UID: \"0cc61754-0414-4c86-84de-250ba68dc560\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q" Jul 30 15:10:44.850986 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.850951 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjxlj\" (UniqueName: \"kubernetes.io/projected/0cc61754-0414-4c86-84de-250ba68dc560-kube-api-access-sjxlj\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q\" (UID: \"0cc61754-0414-4c86-84de-250ba68dc560\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q" Jul 30 15:10:44.851109 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.851009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cc61754-0414-4c86-84de-250ba68dc560-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q\" (UID: \"0cc61754-0414-4c86-84de-250ba68dc560\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q" Jul 30 15:10:44.851109 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.851043 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0cc61754-0414-4c86-84de-250ba68dc560-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q\" (UID: \"0cc61754-0414-4c86-84de-250ba68dc560\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q" Jul 30 15:10:44.851669 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.851647 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0cc61754-0414-4c86-84de-250ba68dc560-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q\" (UID: \"0cc61754-0414-4c86-84de-250ba68dc560\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q" Jul 30 15:10:44.853310 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.853294 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cc61754-0414-4c86-84de-250ba68dc560-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q\" (UID: \"0cc61754-0414-4c86-84de-250ba68dc560\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q" Jul 30 15:10:44.860816 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.860773 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjxlj\" (UniqueName: \"kubernetes.io/projected/0cc61754-0414-4c86-84de-250ba68dc560-kube-api-access-sjxlj\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q\" (UID: \"0cc61754-0414-4c86-84de-250ba68dc560\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q" Jul 30 15:10:44.946871 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:44.946848 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q" Jul 30 15:10:45.052528 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:45.052494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:45.052685 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:45.052544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:45.052685 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:45.052654 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 15:10:45.052807 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:45.052723 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle podName:04216c4f-4a2e-4c3b-a254-b2c11ad44244 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:46.052703896 +0000 UTC m=+73.287834726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle") pod "router-default-5c8b769f9f-tfnst" (UID: "04216c4f-4a2e-4c3b-a254-b2c11ad44244") : configmap references non-existent config key: service-ca.crt Jul 30 15:10:45.052807 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:45.052747 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs podName:04216c4f-4a2e-4c3b-a254-b2c11ad44244 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:46.052736755 +0000 UTC m=+73.287867580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs") pod "router-default-5c8b769f9f-tfnst" (UID: "04216c4f-4a2e-4c3b-a254-b2c11ad44244") : secret "router-metrics-certs-default" not found Jul 30 15:10:45.091528 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:45.091501 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q"] Jul 30 15:10:45.094593 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:10:45.094568 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc61754_0414_4c86_84de_250ba68dc560.slice/crio-f54f55a30b20a8df50b6ec7da8c7550cd36b828988a92eb5fa8435177e93ae91 WatchSource:0}: Error finding container f54f55a30b20a8df50b6ec7da8c7550cd36b828988a92eb5fa8435177e93ae91: Status 404 returned error can't find the container with id f54f55a30b20a8df50b6ec7da8c7550cd36b828988a92eb5fa8435177e93ae91 Jul 30 15:10:45.446330 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:45.446291 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q" event={"ID":"0cc61754-0414-4c86-84de-250ba68dc560","Type":"ContainerStarted","Data":"f54f55a30b20a8df50b6ec7da8c7550cd36b828988a92eb5fa8435177e93ae91"} Jul 30 15:10:46.061006 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:46.060960 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:46.061193 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:46.061017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:46.061193 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:46.061100 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 15:10:46.061193 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:46.061167 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs podName:04216c4f-4a2e-4c3b-a254-b2c11ad44244 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:48.061148774 +0000 UTC m=+75.296279603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs") pod "router-default-5c8b769f9f-tfnst" (UID: "04216c4f-4a2e-4c3b-a254-b2c11ad44244") : secret "router-metrics-certs-default" not found Jul 30 15:10:46.061193 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:46.061187 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle podName:04216c4f-4a2e-4c3b-a254-b2c11ad44244 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:48.061178598 +0000 UTC m=+75.296309459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle") pod "router-default-5c8b769f9f-tfnst" (UID: "04216c4f-4a2e-4c3b-a254-b2c11ad44244") : configmap references non-existent config key: service-ca.crt Jul 30 15:10:48.076895 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:48.076786 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:48.076895 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:48.076850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:48.077394 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:48.076939 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 15:10:48.077394 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:48.076992 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle podName:04216c4f-4a2e-4c3b-a254-b2c11ad44244 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:52.076972738 +0000 UTC m=+79.312103562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle") pod "router-default-5c8b769f9f-tfnst" (UID: "04216c4f-4a2e-4c3b-a254-b2c11ad44244") : configmap references non-existent config key: service-ca.crt Jul 30 15:10:48.077394 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:48.077017 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs podName:04216c4f-4a2e-4c3b-a254-b2c11ad44244 nodeName:}" failed. No retries permitted until 2026-07-30 15:10:52.077006409 +0000 UTC m=+79.312137239 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs") pod "router-default-5c8b769f9f-tfnst" (UID: "04216c4f-4a2e-4c3b-a254-b2c11ad44244") : secret "router-metrics-certs-default" not found Jul 30 15:10:48.454554 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:48.454515 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q" event={"ID":"0cc61754-0414-4c86-84de-250ba68dc560","Type":"ContainerStarted","Data":"09ad6c81a612c465aef682392e3cce3c0398ec4881c5cda54684dc19d0ff3116"} Jul 30 15:10:48.474262 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:48.474216 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q" podStartSLOduration=1.774080084 podStartE2EDuration="4.474202817s" totalImagesPullingTime="2.700122733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:10:48.473179007 +0000 UTC m=+75.708309856" watchObservedRunningTime="2026-07-30 15:10:48.474202817 +0000 UTC m=+75.709333663" Jul 30 15:10:49.052041 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:49.052008 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-tqn4m"] Jul 30 15:10:49.054141 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:49.054116 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tqn4m" Jul 30 15:10:49.056939 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:49.056918 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-665ks\"" Jul 30 15:10:49.057057 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:49.057013 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 30 15:10:49.059414 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:49.059389 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 30 15:10:49.066846 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:49.066820 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-tqn4m"] Jul 30 15:10:49.083319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:49.083295 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r68rq\" (UniqueName: \"kubernetes.io/projected/ed272483-e9f3-48ad-8cb0-a79b10f634dd-kube-api-access-r68rq\") pod \"migrator-6865c9967-tqn4m\" (UID: \"ed272483-e9f3-48ad-8cb0-a79b10f634dd\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tqn4m" Jul 30 15:10:49.184016 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:49.183985 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68rq\" (UniqueName: \"kubernetes.io/projected/ed272483-e9f3-48ad-8cb0-a79b10f634dd-kube-api-access-r68rq\") pod \"migrator-6865c9967-tqn4m\" (UID: \"ed272483-e9f3-48ad-8cb0-a79b10f634dd\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tqn4m" Jul 30 15:10:49.205818 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:49.205778 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r68rq\" (UniqueName: \"kubernetes.io/projected/ed272483-e9f3-48ad-8cb0-a79b10f634dd-kube-api-access-r68rq\") pod \"migrator-6865c9967-tqn4m\" (UID: \"ed272483-e9f3-48ad-8cb0-a79b10f634dd\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tqn4m" Jul 30 15:10:49.362980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:49.362923 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tqn4m" Jul 30 15:10:49.487217 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:49.487190 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-tqn4m"] Jul 30 15:10:49.489953 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:10:49.489927 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded272483_e9f3_48ad_8cb0_a79b10f634dd.slice/crio-bb3a9bcfa3d869902ad062e5f2c4c66e8edecfbb360d88f9328d682854531294 WatchSource:0}: Error finding container bb3a9bcfa3d869902ad062e5f2c4c66e8edecfbb360d88f9328d682854531294: Status 404 returned error can't find the container with id bb3a9bcfa3d869902ad062e5f2c4c66e8edecfbb360d88f9328d682854531294 Jul 30 15:10:50.460506 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.460474 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tqn4m" event={"ID":"ed272483-e9f3-48ad-8cb0-a79b10f634dd","Type":"ContainerStarted","Data":"bb3a9bcfa3d869902ad062e5f2c4c66e8edecfbb360d88f9328d682854531294"} Jul 30 15:10:50.863875 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.863846 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8tgkr"] Jul 30 15:10:50.865959 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.865936 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:50.869026 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.868998 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 30 15:10:50.869159 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.869062 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 30 15:10:50.870446 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.870428 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 30 15:10:50.870576 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.870465 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6v78w\"" Jul 30 15:10:50.870576 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.870481 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 30 15:10:50.883317 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.883290 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8tgkr"] Jul 30 15:10:50.895742 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.895713 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:50.895878 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.895813 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/453cbb90-b8f9-48e1-981d-22a44831433d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:50.895878 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.895870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/453cbb90-b8f9-48e1-981d-22a44831433d-crio-socket\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:50.895993 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.895894 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/453cbb90-b8f9-48e1-981d-22a44831433d-data-volume\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:50.896048 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.896020 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lplzj\" (UniqueName: \"kubernetes.io/projected/453cbb90-b8f9-48e1-981d-22a44831433d-kube-api-access-lplzj\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:50.996565 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.996524 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lplzj\" (UniqueName: \"kubernetes.io/projected/453cbb90-b8f9-48e1-981d-22a44831433d-kube-api-access-lplzj\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:50.996760 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.996580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:50.996760 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.996641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/453cbb90-b8f9-48e1-981d-22a44831433d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:50.996760 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.996697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/453cbb90-b8f9-48e1-981d-22a44831433d-crio-socket\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:50.996760 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.996721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/453cbb90-b8f9-48e1-981d-22a44831433d-data-volume\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:50.997011 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:50.996901 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 15:10:50.997011 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:50.996983 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/453cbb90-b8f9-48e1-981d-22a44831433d-kube-rbac-proxy-cm podName:453cbb90-b8f9-48e1-981d-22a44831433d nodeName:}" failed. No retries permitted until 2026-07-30 15:10:51.496962545 +0000 UTC m=+78.732093379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/453cbb90-b8f9-48e1-981d-22a44831433d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8tgkr" (UID: "453cbb90-b8f9-48e1-981d-22a44831433d") : configmap "kube-rbac-proxy" not found Jul 30 15:10:50.997011 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.996993 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/453cbb90-b8f9-48e1-981d-22a44831433d-crio-socket\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:50.997170 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:50.996999 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 15:10:50.997170 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:50.997072 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls podName:453cbb90-b8f9-48e1-981d-22a44831433d nodeName:}" failed. No retries permitted until 2026-07-30 15:10:51.497058755 +0000 UTC m=+78.732189579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8tgkr" (UID: "453cbb90-b8f9-48e1-981d-22a44831433d") : secret "insights-runtime-extractor-tls" not found Jul 30 15:10:50.997170 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:50.997131 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/453cbb90-b8f9-48e1-981d-22a44831433d-data-volume\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:51.008440 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:51.008419 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lplzj\" (UniqueName: \"kubernetes.io/projected/453cbb90-b8f9-48e1-981d-22a44831433d-kube-api-access-lplzj\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:51.464838 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:51.464739 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tqn4m" event={"ID":"ed272483-e9f3-48ad-8cb0-a79b10f634dd","Type":"ContainerStarted","Data":"79a2ae4f072c88cbd74a22f0bf302f4d8413cf451886c68ee62346191e0746d4"} Jul 30 15:10:51.464838 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:51.464812 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tqn4m" event={"ID":"ed272483-e9f3-48ad-8cb0-a79b10f634dd","Type":"ContainerStarted","Data":"0fde035e8344d7f991999578e9f8c6fb79b4e134950a267cf8a29850ad29d897"} Jul 30 15:10:51.486735 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:51.486695 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tqn4m" podStartSLOduration=0.868008051 podStartE2EDuration="2.486681329s" totalImagesPullingTime="1.618673278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:10:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:10:51.485072869 +0000 UTC m=+78.720203711" watchObservedRunningTime="2026-07-30 15:10:51.486681329 +0000 UTC m=+78.721812176" Jul 30 15:10:51.500516 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:51.500493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:51.500585 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:51.500544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/453cbb90-b8f9-48e1-981d-22a44831433d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:51.500647 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:51.500630 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 15:10:51.500687 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:51.500682 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls podName:453cbb90-b8f9-48e1-981d-22a44831433d nodeName:}" failed. No retries permitted until 2026-07-30 15:10:52.500668824 +0000 UTC m=+79.735799648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8tgkr" (UID: "453cbb90-b8f9-48e1-981d-22a44831433d") : secret "insights-runtime-extractor-tls" not found Jul 30 15:10:51.500745 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:51.500632 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 15:10:51.500816 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:51.500770 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/453cbb90-b8f9-48e1-981d-22a44831433d-kube-rbac-proxy-cm podName:453cbb90-b8f9-48e1-981d-22a44831433d nodeName:}" failed. No retries permitted until 2026-07-30 15:10:52.500748208 +0000 UTC m=+79.735879051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/453cbb90-b8f9-48e1-981d-22a44831433d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8tgkr" (UID: "453cbb90-b8f9-48e1-981d-22a44831433d") : configmap "kube-rbac-proxy" not found Jul 30 15:10:52.105812 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:52.105760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:52.105991 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:52.105931 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle podName:04216c4f-4a2e-4c3b-a254-b2c11ad44244 nodeName:}" failed. No retries permitted until 2026-07-30 15:11:00.105916074 +0000 UTC m=+87.341046898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle") pod "router-default-5c8b769f9f-tfnst" (UID: "04216c4f-4a2e-4c3b-a254-b2c11ad44244") : configmap references non-existent config key: service-ca.crt Jul 30 15:10:52.105991 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:52.105975 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:10:52.106096 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:52.106051 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 15:10:52.106096 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:52.106091 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs podName:04216c4f-4a2e-4c3b-a254-b2c11ad44244 nodeName:}" failed. No retries permitted until 2026-07-30 15:11:00.106082992 +0000 UTC m=+87.341213817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs") pod "router-default-5c8b769f9f-tfnst" (UID: "04216c4f-4a2e-4c3b-a254-b2c11ad44244") : secret "router-metrics-certs-default" not found Jul 30 15:10:52.509545 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:52.509517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:52.509981 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:52.509568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/453cbb90-b8f9-48e1-981d-22a44831433d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:52.509981 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:52.509636 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 15:10:52.509981 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:52.509673 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 15:10:52.509981 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:52.509689 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/453cbb90-b8f9-48e1-981d-22a44831433d-kube-rbac-proxy-cm podName:453cbb90-b8f9-48e1-981d-22a44831433d nodeName:}" failed. No retries permitted until 2026-07-30 15:10:54.509672225 +0000 UTC m=+81.744803066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/453cbb90-b8f9-48e1-981d-22a44831433d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8tgkr" (UID: "453cbb90-b8f9-48e1-981d-22a44831433d") : configmap "kube-rbac-proxy" not found Jul 30 15:10:52.509981 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:52.509747 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls podName:453cbb90-b8f9-48e1-981d-22a44831433d nodeName:}" failed. No retries permitted until 2026-07-30 15:10:54.509731536 +0000 UTC m=+81.744862361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8tgkr" (UID: "453cbb90-b8f9-48e1-981d-22a44831433d") : secret "insights-runtime-extractor-tls" not found Jul 30 15:10:53.050246 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:53.050211 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cvbfc_c35ba36d-8199-471f-99df-70cc5ba0c05e/dns-node-resolver/0.log" Jul 30 15:10:54.249573 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:54.249541 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g8lsb_ec9478e9-d9e0-4ea7-9412-e36fc1a305c7/node-ca/0.log" Jul 30 15:10:54.524451 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:54.524371 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:54.524598 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:54.524500 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 15:10:54.524598 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:54.524518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/453cbb90-b8f9-48e1-981d-22a44831433d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:54.524598 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:54.524553 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls podName:453cbb90-b8f9-48e1-981d-22a44831433d nodeName:}" failed. No retries permitted until 2026-07-30 15:10:58.524538896 +0000 UTC m=+85.759669719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8tgkr" (UID: "453cbb90-b8f9-48e1-981d-22a44831433d") : secret "insights-runtime-extractor-tls" not found Jul 30 15:10:54.524598 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:54.524584 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 15:10:54.524775 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:54.524641 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/453cbb90-b8f9-48e1-981d-22a44831433d-kube-rbac-proxy-cm podName:453cbb90-b8f9-48e1-981d-22a44831433d nodeName:}" failed. No retries permitted until 2026-07-30 15:10:58.524624778 +0000 UTC m=+85.759755622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/453cbb90-b8f9-48e1-981d-22a44831433d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8tgkr" (UID: "453cbb90-b8f9-48e1-981d-22a44831433d") : configmap "kube-rbac-proxy" not found Jul 30 15:10:54.849309 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:54.849242 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ksbjt_d5098562-bcb2-4825-8429-184d57da5659/serve-healthcheck-canary/0.log" Jul 30 15:10:55.249333 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:55.249297 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-tqn4m_ed272483-e9f3-48ad-8cb0-a79b10f634dd/migrator/0.log" Jul 30 15:10:55.448561 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:55.448527 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-tqn4m_ed272483-e9f3-48ad-8cb0-a79b10f634dd/graceful-termination/0.log" Jul 30 15:10:55.650820 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:55.650781 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q_0cc61754-0414-4c86-84de-250ba68dc560/kube-storage-version-migrator-operator/0.log" Jul 30 15:10:58.558439 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:58.558402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/453cbb90-b8f9-48e1-981d-22a44831433d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:58.558891 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:10:58.558502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:10:58.558891 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:58.558517 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 15:10:58.558891 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:58.558574 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/453cbb90-b8f9-48e1-981d-22a44831433d-kube-rbac-proxy-cm podName:453cbb90-b8f9-48e1-981d-22a44831433d nodeName:}" failed. No retries permitted until 2026-07-30 15:11:06.558558008 +0000 UTC m=+93.793688852 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/453cbb90-b8f9-48e1-981d-22a44831433d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8tgkr" (UID: "453cbb90-b8f9-48e1-981d-22a44831433d") : configmap "kube-rbac-proxy" not found Jul 30 15:10:58.558891 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:58.558633 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 15:10:58.558891 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:10:58.558689 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls podName:453cbb90-b8f9-48e1-981d-22a44831433d nodeName:}" failed. No retries permitted until 2026-07-30 15:11:06.558677171 +0000 UTC m=+93.793807995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8tgkr" (UID: "453cbb90-b8f9-48e1-981d-22a44831433d") : secret "insights-runtime-extractor-tls" not found Jul 30 15:11:00.171242 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:00.171196 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:11:00.171242 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:00.171240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:11:00.171660 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:11:00.171341 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 15:11:00.171660 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:11:00.171397 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs podName:04216c4f-4a2e-4c3b-a254-b2c11ad44244 nodeName:}" failed. No retries permitted until 2026-07-30 15:11:16.171382916 +0000 UTC m=+103.406513741 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs") pod "router-default-5c8b769f9f-tfnst" (UID: "04216c4f-4a2e-4c3b-a254-b2c11ad44244") : secret "router-metrics-certs-default" not found Jul 30 15:11:00.171660 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:11:00.171412 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle podName:04216c4f-4a2e-4c3b-a254-b2c11ad44244 nodeName:}" failed. No retries permitted until 2026-07-30 15:11:16.171404044 +0000 UTC m=+103.406534867 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle") pod "router-default-5c8b769f9f-tfnst" (UID: "04216c4f-4a2e-4c3b-a254-b2c11ad44244") : configmap references non-existent config key: service-ca.crt Jul 30 15:11:06.620133 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:06.620091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:11:06.620623 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:06.620154 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/453cbb90-b8f9-48e1-981d-22a44831433d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:11:06.620623 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:11:06.620249 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 15:11:06.620623 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:11:06.620303 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls podName:453cbb90-b8f9-48e1-981d-22a44831433d nodeName:}" failed. No retries permitted until 2026-07-30 15:11:22.620289183 +0000 UTC m=+109.855420009 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8tgkr" (UID: "453cbb90-b8f9-48e1-981d-22a44831433d") : secret "insights-runtime-extractor-tls" not found Jul 30 15:11:06.620743 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:06.620633 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/453cbb90-b8f9-48e1-981d-22a44831433d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:11:12.059876 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:12.059837 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:11:12.059876 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:12.059879 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls\") pod \"image-registry-f875d6977-b47mk\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:11:12.060266 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:11:12.059981 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 15:11:12.060266 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:11:12.059997 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f875d6977-b47mk: secret "image-registry-tls" not found Jul 30 15:11:12.060266 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:11:12.060041 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls podName:406028a3-cf3c-44ee-83da-39aae2661e87 nodeName:}" failed. No retries permitted until 2026-07-30 15:12:16.060028173 +0000 UTC m=+163.295158998 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls") pod "image-registry-f875d6977-b47mk" (UID: "406028a3-cf3c-44ee-83da-39aae2661e87") : secret "image-registry-tls" not found Jul 30 15:11:12.060266 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:11:12.059980 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 15:11:12.060266 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:11:12.060114 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls podName:829b9cdd-657c-4f62-8a83-813c43eefa09 nodeName:}" failed. No retries permitted until 2026-07-30 15:12:16.060100369 +0000 UTC m=+163.295231193 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls") pod "dns-default-gq5ph" (UID: "829b9cdd-657c-4f62-8a83-813c43eefa09") : secret "dns-default-metrics-tls" not found Jul 30 15:11:15.448644 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:15.448613 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pc6xx" Jul 30 15:11:16.189136 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:16.189096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:11:16.189136 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:16.189139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:11:16.189326 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:11:16.189249 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 15:11:16.189326 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:11:16.189278 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle podName:04216c4f-4a2e-4c3b-a254-b2c11ad44244 nodeName:}" failed. No retries permitted until 2026-07-30 15:11:48.189264953 +0000 UTC m=+135.424395776 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle") pod "router-default-5c8b769f9f-tfnst" (UID: "04216c4f-4a2e-4c3b-a254-b2c11ad44244") : configmap references non-existent config key: service-ca.crt Jul 30 15:11:16.189326 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:11:16.189315 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs podName:04216c4f-4a2e-4c3b-a254-b2c11ad44244 nodeName:}" failed. No retries permitted until 2026-07-30 15:11:48.189300507 +0000 UTC m=+135.424431330 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs") pod "router-default-5c8b769f9f-tfnst" (UID: "04216c4f-4a2e-4c3b-a254-b2c11ad44244") : secret "router-metrics-certs-default" not found Jul 30 15:11:22.631668 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:22.631631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:11:22.632147 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:11:22.631766 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 15:11:22.632147 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:11:22.631836 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls podName:453cbb90-b8f9-48e1-981d-22a44831433d nodeName:}" failed. No retries permitted until 2026-07-30 15:11:54.631821901 +0000 UTC m=+141.866952725 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8tgkr" (UID: "453cbb90-b8f9-48e1-981d-22a44831433d") : secret "insights-runtime-extractor-tls" not found Jul 30 15:11:42.968628 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:42.968580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs\") pod \"network-metrics-daemon-rsf9b\" (UID: \"a8cf4460-ac48-44e7-9a6e-64ff431ca403\") " pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:11:42.971058 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:42.971033 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cf4460-ac48-44e7-9a6e-64ff431ca403-metrics-certs\") pod \"network-metrics-daemon-rsf9b\" (UID: \"a8cf4460-ac48-44e7-9a6e-64ff431ca403\") " pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:11:43.008386 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:43.008362 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kvhp8\"" Jul 30 15:11:43.016642 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:43.016625 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsf9b" Jul 30 15:11:43.142222 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:43.142191 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rsf9b"] Jul 30 15:11:43.145010 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:11:43.144986 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cf4460_ac48_44e7_9a6e_64ff431ca403.slice/crio-923f3ce3dd2e9f9b008b16ee2d99f4bcbe65407ec82eb3d23d18aabf00e954a5 WatchSource:0}: Error finding container 923f3ce3dd2e9f9b008b16ee2d99f4bcbe65407ec82eb3d23d18aabf00e954a5: Status 404 returned error can't find the container with id 923f3ce3dd2e9f9b008b16ee2d99f4bcbe65407ec82eb3d23d18aabf00e954a5 Jul 30 15:11:43.612828 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:43.612751 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rsf9b" event={"ID":"a8cf4460-ac48-44e7-9a6e-64ff431ca403","Type":"ContainerStarted","Data":"923f3ce3dd2e9f9b008b16ee2d99f4bcbe65407ec82eb3d23d18aabf00e954a5"} Jul 30 15:11:44.618046 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:44.618006 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rsf9b" event={"ID":"a8cf4460-ac48-44e7-9a6e-64ff431ca403","Type":"ContainerStarted","Data":"457fb5c8ff01f5849605e1f0a99b104f9a8bdc2078926aef561aa22df70e22ad"} Jul 30 15:11:44.618510 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:44.618059 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rsf9b" event={"ID":"a8cf4460-ac48-44e7-9a6e-64ff431ca403","Type":"ContainerStarted","Data":"63a0a53d4204552c715a37a3921a088ec87bb61e8efc501682298c414700b03b"} Jul 30 15:11:44.636126 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:44.636067 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rsf9b" podStartSLOduration=130.67197954 podStartE2EDuration="2m11.636048654s" totalImagesPullingTime="964.069114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:11:44.635736694 +0000 UTC m=+131.870867539" watchObservedRunningTime="2026-07-30 15:11:44.636048654 +0000 UTC m=+131.871179502" Jul 30 15:11:48.214059 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:48.214028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:11:48.214059 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:48.214061 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:11:48.214594 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:48.214576 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04216c4f-4a2e-4c3b-a254-b2c11ad44244-service-ca-bundle\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:11:48.216444 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:48.216422 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04216c4f-4a2e-4c3b-a254-b2c11ad44244-metrics-certs\") pod \"router-default-5c8b769f9f-tfnst\" (UID: \"04216c4f-4a2e-4c3b-a254-b2c11ad44244\") " pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:11:48.324256 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:48.324226 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xtsqs\"" Jul 30 15:11:48.331826 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:48.331793 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:11:48.459482 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:48.459451 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c8b769f9f-tfnst"] Jul 30 15:11:48.462273 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:11:48.462249 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04216c4f_4a2e_4c3b_a254_b2c11ad44244.slice/crio-0eb53d8ac4ea36acf39b317705b41bf7cd786014a76130c3fae1a3da6c64615b WatchSource:0}: Error finding container 0eb53d8ac4ea36acf39b317705b41bf7cd786014a76130c3fae1a3da6c64615b: Status 404 returned error can't find the container with id 0eb53d8ac4ea36acf39b317705b41bf7cd786014a76130c3fae1a3da6c64615b Jul 30 15:11:48.630567 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:48.630529 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c8b769f9f-tfnst" event={"ID":"04216c4f-4a2e-4c3b-a254-b2c11ad44244","Type":"ContainerStarted","Data":"e2b176df340340cea0042c0978bfc06ad4112c296557ab8c824f7c846d4d1389"} Jul 30 15:11:48.630725 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:48.630604 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c8b769f9f-tfnst" event={"ID":"04216c4f-4a2e-4c3b-a254-b2c11ad44244","Type":"ContainerStarted","Data":"0eb53d8ac4ea36acf39b317705b41bf7cd786014a76130c3fae1a3da6c64615b"} Jul 30 15:11:48.653486 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:48.653350 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5c8b769f9f-tfnst" podStartSLOduration=64.653334622 podStartE2EDuration="1m4.653334622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:11:48.653101865 +0000 UTC m=+135.888232711" watchObservedRunningTime="2026-07-30 15:11:48.653334622 +0000 UTC m=+135.888465469" Jul 30 15:11:49.332298 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:49.332261 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:11:49.334841 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:49.334816 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:11:49.634319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:49.634287 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:11:49.635463 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:49.635445 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5c8b769f9f-tfnst" Jul 30 15:11:51.691518 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:51.691485 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9t29f"] Jul 30 15:11:51.693521 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:51.693506 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9t29f" Jul 30 15:11:51.697606 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:51.697586 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 30 15:11:51.697725 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:51.697610 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4vgqc\"" Jul 30 15:11:51.706289 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:51.706263 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9t29f"] Jul 30 15:11:51.738470 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:51.738443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bb7ed78c-9c26-4a78-b39e-ec702a794f7e-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9t29f\" (UID: \"bb7ed78c-9c26-4a78-b39e-ec702a794f7e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9t29f" Jul 30 15:11:51.839686 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:51.839655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bb7ed78c-9c26-4a78-b39e-ec702a794f7e-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9t29f\" (UID: \"bb7ed78c-9c26-4a78-b39e-ec702a794f7e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9t29f" Jul 30 15:11:51.841950 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:51.841928 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bb7ed78c-9c26-4a78-b39e-ec702a794f7e-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9t29f\" (UID: \"bb7ed78c-9c26-4a78-b39e-ec702a794f7e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9t29f" Jul 30 15:11:52.002526 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:52.002418 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9t29f" Jul 30 15:11:52.125949 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:52.125919 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9t29f"] Jul 30 15:11:52.129155 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:11:52.129133 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7ed78c_9c26_4a78_b39e_ec702a794f7e.slice/crio-41d81c4d0378154873e98e0b30a82e416e00b664e8b6f080fc97370f28f52308 WatchSource:0}: Error finding container 41d81c4d0378154873e98e0b30a82e416e00b664e8b6f080fc97370f28f52308: Status 404 returned error can't find the container with id 41d81c4d0378154873e98e0b30a82e416e00b664e8b6f080fc97370f28f52308 Jul 30 15:11:52.644434 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:52.644395 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9t29f" event={"ID":"bb7ed78c-9c26-4a78-b39e-ec702a794f7e","Type":"ContainerStarted","Data":"41d81c4d0378154873e98e0b30a82e416e00b664e8b6f080fc97370f28f52308"} Jul 30 15:11:53.649225 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:53.649190 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9t29f" event={"ID":"bb7ed78c-9c26-4a78-b39e-ec702a794f7e","Type":"ContainerStarted","Data":"90e08bde8c11a38e719c2a3e219eabcc219dd8881bf0477054d8c6594ccd046e"} Jul 30 15:11:53.649675 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:53.649383 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9t29f" Jul 30 15:11:53.653633 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:53.653604 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9t29f" Jul 30 15:11:53.677218 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:53.677172 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9t29f" podStartSLOduration=1.598749097 podStartE2EDuration="2.677159311s" totalImagesPullingTime="1.078410214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:11:53.676688547 +0000 UTC m=+140.911819393" watchObservedRunningTime="2026-07-30 15:11:53.677159311 +0000 UTC m=+140.912290157" Jul 30 15:11:54.655887 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.655856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:11:54.658089 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.658068 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/453cbb90-b8f9-48e1-981d-22a44831433d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8tgkr\" (UID: \"453cbb90-b8f9-48e1-981d-22a44831433d\") " pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:11:54.770528 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.770502 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-ldkrv"] Jul 30 15:11:54.772527 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.772512 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-ldkrv" Jul 30 15:11:54.776019 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.775997 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 30 15:11:54.778959 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.778938 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 30 15:11:54.779040 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.778938 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6v78w\"" Jul 30 15:11:54.779040 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.778940 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 30 15:11:54.779040 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.779002 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-drqx9\"" Jul 30 15:11:54.779937 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.779924 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 30 15:11:54.781570 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.781556 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 30 15:11:54.785664 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.785651 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8tgkr" Jul 30 15:11:54.795565 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.795543 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-ldkrv"] Jul 30 15:11:54.859153 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.859114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk8lp\" (UniqueName: \"kubernetes.io/projected/3ab8732b-7ec2-4dd1-bd57-4c2907e2394e-kube-api-access-fk8lp\") pod \"prometheus-operator-84d69f794d-ldkrv\" (UID: \"3ab8732b-7ec2-4dd1-bd57-4c2907e2394e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ldkrv" Jul 30 15:11:54.859432 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.859411 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ab8732b-7ec2-4dd1-bd57-4c2907e2394e-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-ldkrv\" (UID: \"3ab8732b-7ec2-4dd1-bd57-4c2907e2394e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ldkrv" Jul 30 15:11:54.859586 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.859570 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ab8732b-7ec2-4dd1-bd57-4c2907e2394e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-ldkrv\" (UID: \"3ab8732b-7ec2-4dd1-bd57-4c2907e2394e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ldkrv" Jul 30 15:11:54.859721 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.859708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ab8732b-7ec2-4dd1-bd57-4c2907e2394e-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-ldkrv\" (UID: \"3ab8732b-7ec2-4dd1-bd57-4c2907e2394e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ldkrv" Jul 30 15:11:54.932892 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.932860 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8tgkr"] Jul 30 15:11:54.935107 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:11:54.935070 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453cbb90_b8f9_48e1_981d_22a44831433d.slice/crio-a28ac7632b4304a7ca8678c0d0b433467306442ed6f714284141ba3588b4d4da WatchSource:0}: Error finding container a28ac7632b4304a7ca8678c0d0b433467306442ed6f714284141ba3588b4d4da: Status 404 returned error can't find the container with id a28ac7632b4304a7ca8678c0d0b433467306442ed6f714284141ba3588b4d4da Jul 30 15:11:54.961074 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.961052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ab8732b-7ec2-4dd1-bd57-4c2907e2394e-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-ldkrv\" (UID: \"3ab8732b-7ec2-4dd1-bd57-4c2907e2394e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ldkrv" Jul 30 15:11:54.961172 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.961081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ab8732b-7ec2-4dd1-bd57-4c2907e2394e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-ldkrv\" (UID: \"3ab8732b-7ec2-4dd1-bd57-4c2907e2394e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ldkrv" Jul 30 15:11:54.961172 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.961102 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ab8732b-7ec2-4dd1-bd57-4c2907e2394e-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-ldkrv\" (UID: \"3ab8732b-7ec2-4dd1-bd57-4c2907e2394e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ldkrv" Jul 30 15:11:54.961248 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.961191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk8lp\" (UniqueName: \"kubernetes.io/projected/3ab8732b-7ec2-4dd1-bd57-4c2907e2394e-kube-api-access-fk8lp\") pod \"prometheus-operator-84d69f794d-ldkrv\" (UID: \"3ab8732b-7ec2-4dd1-bd57-4c2907e2394e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ldkrv" Jul 30 15:11:54.961704 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.961683 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ab8732b-7ec2-4dd1-bd57-4c2907e2394e-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-ldkrv\" (UID: \"3ab8732b-7ec2-4dd1-bd57-4c2907e2394e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ldkrv" Jul 30 15:11:54.963329 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.963311 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ab8732b-7ec2-4dd1-bd57-4c2907e2394e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-ldkrv\" (UID: \"3ab8732b-7ec2-4dd1-bd57-4c2907e2394e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ldkrv" Jul 30 15:11:54.963486 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.963469 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ab8732b-7ec2-4dd1-bd57-4c2907e2394e-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-ldkrv\" (UID: \"3ab8732b-7ec2-4dd1-bd57-4c2907e2394e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ldkrv" Jul 30 15:11:54.974259 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:54.974240 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk8lp\" (UniqueName: \"kubernetes.io/projected/3ab8732b-7ec2-4dd1-bd57-4c2907e2394e-kube-api-access-fk8lp\") pod \"prometheus-operator-84d69f794d-ldkrv\" (UID: \"3ab8732b-7ec2-4dd1-bd57-4c2907e2394e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ldkrv" Jul 30 15:11:55.081470 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:55.081439 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-ldkrv" Jul 30 15:11:55.208674 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:55.208649 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-ldkrv"] Jul 30 15:11:55.210343 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:11:55.210316 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab8732b_7ec2_4dd1_bd57_4c2907e2394e.slice/crio-34f40f9aa8a936178152c2e0ebdd10dcb40cee4f83a02e1aabd119b4063ebcbc WatchSource:0}: Error finding container 34f40f9aa8a936178152c2e0ebdd10dcb40cee4f83a02e1aabd119b4063ebcbc: Status 404 returned error can't find the container with id 34f40f9aa8a936178152c2e0ebdd10dcb40cee4f83a02e1aabd119b4063ebcbc Jul 30 15:11:55.662264 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:55.662188 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-ldkrv" event={"ID":"3ab8732b-7ec2-4dd1-bd57-4c2907e2394e","Type":"ContainerStarted","Data":"34f40f9aa8a936178152c2e0ebdd10dcb40cee4f83a02e1aabd119b4063ebcbc"} Jul 30 15:11:55.663859 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:55.663829 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8tgkr" event={"ID":"453cbb90-b8f9-48e1-981d-22a44831433d","Type":"ContainerStarted","Data":"a28ac7632b4304a7ca8678c0d0b433467306442ed6f714284141ba3588b4d4da"} Jul 30 15:11:56.670682 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:56.670598 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-ldkrv" event={"ID":"3ab8732b-7ec2-4dd1-bd57-4c2907e2394e","Type":"ContainerStarted","Data":"573e07c57a241b5043daf1c1b28eb43bee134992ccc92d757fb8ea5e52ca82f8"} Jul 30 15:11:57.676267 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:57.676199 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-ldkrv" event={"ID":"3ab8732b-7ec2-4dd1-bd57-4c2907e2394e","Type":"ContainerStarted","Data":"f62d83b20ff3701819bec43d5a187b9261afb60630dc96645889357671b47938"} Jul 30 15:11:57.711832 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:57.711755 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-ldkrv" podStartSLOduration=2.36269519 podStartE2EDuration="3.711734753s" totalImagesPullingTime="1.349039563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:11:57.711513859 +0000 UTC m=+144.946644716" watchObservedRunningTime="2026-07-30 15:11:57.711734753 +0000 UTC m=+144.946865600" Jul 30 15:11:59.676315 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.676279 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg"] Jul 30 15:11:59.679001 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.678983 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.685267 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.685242 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 30 15:11:59.686329 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.686309 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fbf84\"" Jul 30 15:11:59.686447 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.686362 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 30 15:11:59.686447 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.686435 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 30 15:11:59.698830 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.698791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-pdxgg\" (UID: \"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.698943 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.698921 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-pdxgg\" (UID: \"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.698989 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.698968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-pdxgg\" (UID: \"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.699036 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.699000 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-pdxgg\" (UID: \"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.699084 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.699042 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-pdxgg\" (UID: \"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.699130 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.699089 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbm4q\" (UniqueName: \"kubernetes.io/projected/ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9-kube-api-access-vbm4q\") pod \"kube-state-metrics-56d9f944b4-pdxgg\" (UID: \"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.729997 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.729971 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5clt4"] Jul 30 15:11:59.732529 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.732509 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.735252 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.735224 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg"] Jul 30 15:11:59.736063 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.736041 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 30 15:11:59.736181 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.736045 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tpj8t\"" Jul 30 15:11:59.736841 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.736518 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 30 15:11:59.736841 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.736671 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 30 15:11:59.799967 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.799933 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e653e12c-86c8-4cca-897f-a7677c143087-node-exporter-tls\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.800135 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.799988 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-pdxgg\" (UID: \"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.800135 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.800023 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e653e12c-86c8-4cca-897f-a7677c143087-node-exporter-accelerators-collector-config\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.800135 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.800104 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e653e12c-86c8-4cca-897f-a7677c143087-metrics-client-ca\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.800320 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.800137 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e653e12c-86c8-4cca-897f-a7677c143087-node-exporter-wtmp\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.800320 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.800173 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbm4q\" (UniqueName: \"kubernetes.io/projected/ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9-kube-api-access-vbm4q\") pod \"kube-state-metrics-56d9f944b4-pdxgg\" (UID: \"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.800320 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.800207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-pdxgg\" (UID: \"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.800320 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.800233 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e653e12c-86c8-4cca-897f-a7677c143087-node-exporter-textfile\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.800320 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.800264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e653e12c-86c8-4cca-897f-a7677c143087-sys\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.800320 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.800316 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e653e12c-86c8-4cca-897f-a7677c143087-root\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.800603 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.800359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-pdxgg\" (UID: \"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.800603 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.800391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktdxx\" (UniqueName: \"kubernetes.io/projected/e653e12c-86c8-4cca-897f-a7677c143087-kube-api-access-ktdxx\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.800603 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.800422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-pdxgg\" (UID: \"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.800603 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.800453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-pdxgg\" (UID: \"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.800603 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.800481 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e653e12c-86c8-4cca-897f-a7677c143087-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.801140 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.801094 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-pdxgg\" (UID: \"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.801415 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.801366 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-pdxgg\" (UID: \"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.801534 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.801450 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-pdxgg\" (UID: \"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.803149 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.803125 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-pdxgg\" (UID: \"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.803698 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.803670 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-pdxgg\" (UID: \"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.816088 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.816070 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbm4q\" (UniqueName: \"kubernetes.io/projected/ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9-kube-api-access-vbm4q\") pod \"kube-state-metrics-56d9f944b4-pdxgg\" (UID: \"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:11:59.901301 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.901261 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e653e12c-86c8-4cca-897f-a7677c143087-sys\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.901478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.901332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e653e12c-86c8-4cca-897f-a7677c143087-root\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.901478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.901379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktdxx\" (UniqueName: \"kubernetes.io/projected/e653e12c-86c8-4cca-897f-a7677c143087-kube-api-access-ktdxx\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.901478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.901407 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e653e12c-86c8-4cca-897f-a7677c143087-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.901478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.901401 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e653e12c-86c8-4cca-897f-a7677c143087-sys\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.901478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.901434 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e653e12c-86c8-4cca-897f-a7677c143087-root\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.901478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.901448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e653e12c-86c8-4cca-897f-a7677c143087-node-exporter-tls\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.901783 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.901505 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e653e12c-86c8-4cca-897f-a7677c143087-node-exporter-accelerators-collector-config\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.901783 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.901548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e653e12c-86c8-4cca-897f-a7677c143087-metrics-client-ca\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.901783 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.901575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e653e12c-86c8-4cca-897f-a7677c143087-node-exporter-wtmp\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.901783 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.901607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e653e12c-86c8-4cca-897f-a7677c143087-node-exporter-textfile\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.902023 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.901832 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e653e12c-86c8-4cca-897f-a7677c143087-node-exporter-wtmp\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.902023 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.901973 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e653e12c-86c8-4cca-897f-a7677c143087-node-exporter-textfile\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.902310 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.902263 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e653e12c-86c8-4cca-897f-a7677c143087-node-exporter-accelerators-collector-config\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.902310 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.902297 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e653e12c-86c8-4cca-897f-a7677c143087-metrics-client-ca\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.904375 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.904356 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e653e12c-86c8-4cca-897f-a7677c143087-node-exporter-tls\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.904555 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.904531 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e653e12c-86c8-4cca-897f-a7677c143087-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.926774 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.926717 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktdxx\" (UniqueName: \"kubernetes.io/projected/e653e12c-86c8-4cca-897f-a7677c143087-kube-api-access-ktdxx\") pod \"node-exporter-5clt4\" (UID: \"e653e12c-86c8-4cca-897f-a7677c143087\") " pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:11:59.993889 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:11:59.993869 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" Jul 30 15:12:00.044334 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.043014 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5clt4" Jul 30 15:12:00.056661 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:12:00.056623 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode653e12c_86c8_4cca_897f_a7677c143087.slice/crio-17b10f837e1d021ee0edecfe91e08731996c55a8636000fdcb89c01a7ac802c5 WatchSource:0}: Error finding container 17b10f837e1d021ee0edecfe91e08731996c55a8636000fdcb89c01a7ac802c5: Status 404 returned error can't find the container with id 17b10f837e1d021ee0edecfe91e08731996c55a8636000fdcb89c01a7ac802c5 Jul 30 15:12:00.172290 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.172256 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg"] Jul 30 15:12:00.175050 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:12:00.175025 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5fd0b1_70ff_46cb_bc48_1a6dc61a75b9.slice/crio-07ed9bc182d40d6888859808d881588efa78b6fcfbdfdd5c27af35195f7d7420 WatchSource:0}: Error finding container 07ed9bc182d40d6888859808d881588efa78b6fcfbdfdd5c27af35195f7d7420: Status 404 returned error can't find the container with id 07ed9bc182d40d6888859808d881588efa78b6fcfbdfdd5c27af35195f7d7420 Jul 30 15:12:00.623715 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.623684 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 15:12:00.626830 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.626772 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.633716 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.633188 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 30 15:12:00.636893 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.636473 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qg6rg\"" Jul 30 15:12:00.636893 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.636698 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 30 15:12:00.642317 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.642057 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 30 15:12:00.642317 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.642162 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 30 15:12:00.643815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.643439 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 30 15:12:00.643815 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.643648 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 30 15:12:00.644473 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.644304 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 30 15:12:00.651458 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.650572 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 30 15:12:00.670594 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.670563 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 30 15:12:00.696414 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.696377 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8tgkr" event={"ID":"453cbb90-b8f9-48e1-981d-22a44831433d","Type":"ContainerStarted","Data":"553b54f52d69852d2368adf9d7b8dfee66a1244b6df48896dbefe77dd349290d"} Jul 30 15:12:00.699331 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.699259 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" event={"ID":"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9","Type":"ContainerStarted","Data":"07ed9bc182d40d6888859808d881588efa78b6fcfbdfdd5c27af35195f7d7420"} Jul 30 15:12:00.700451 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.700367 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 15:12:00.701568 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.701502 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5clt4" event={"ID":"e653e12c-86c8-4cca-897f-a7677c143087","Type":"ContainerStarted","Data":"17b10f837e1d021ee0edecfe91e08731996c55a8636000fdcb89c01a7ac802c5"} Jul 30 15:12:00.708825 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.708250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76nsr\" (UniqueName: \"kubernetes.io/projected/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-kube-api-access-76nsr\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.708825 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.708293 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.708825 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.708321 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.708825 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.708359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.708825 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.708385 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.708825 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.708413 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.708825 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.708457 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.708825 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.708482 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.708825 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.708537 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.708825 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.708566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-web-config\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.708825 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.708588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.708825 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.708650 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-config-volume\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.708825 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.708678 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-config-out\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.809311 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.809274 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.809493 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.809324 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.809493 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.809352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.809493 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.809387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.809493 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.809413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.809493 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.809466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.809493 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.809494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-web-config\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.810266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.809515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.810266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.809572 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-config-volume\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.810266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.809597 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-config-out\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.810266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.809637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76nsr\" (UniqueName: \"kubernetes.io/projected/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-kube-api-access-76nsr\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.810266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.809665 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.810266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.809690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.810669 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.810495 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.813088 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.811336 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.813088 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:12:00.811525 2579 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 30 15:12:00.813088 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:12:00.811592 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-main-tls podName:8fe47b82-3ab6-4505-99bf-b6c5bb503a0a nodeName:}" failed. No retries permitted until 2026-07-30 15:12:01.311571738 +0000 UTC m=+148.546702572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a") : secret "alertmanager-main-tls" not found Jul 30 15:12:00.813088 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.813024 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.813879 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.813788 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-web-config\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.814067 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.814045 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.815504 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.815192 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.815504 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.815237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.816236 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.816208 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-config-volume\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.817314 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.817273 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.817881 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.817838 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.819790 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.819767 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-config-out\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:00.832201 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:00.832174 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76nsr\" (UniqueName: \"kubernetes.io/projected/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-kube-api-access-76nsr\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:01.316001 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:01.315967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:01.319093 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:01.319068 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:01.543231 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:01.543153 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:12:01.707195 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:01.707155 2579 generic.go:356] "Generic (PLEG): container finished" podID="e653e12c-86c8-4cca-897f-a7677c143087" containerID="1f80c24e8de32824dc88596b65da214c3dea1bc12d7b61e1f2b454d21a26b92d" exitCode=0 Jul 30 15:12:01.707666 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:01.707242 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5clt4" event={"ID":"e653e12c-86c8-4cca-897f-a7677c143087","Type":"ContainerDied","Data":"1f80c24e8de32824dc88596b65da214c3dea1bc12d7b61e1f2b454d21a26b92d"} Jul 30 15:12:03.328352 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:03.328303 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 15:12:03.825416 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:12:03.825384 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe47b82_3ab6_4505_99bf_b6c5bb503a0a.slice/crio-f99ac00ea5b99ab2ec8b39b3bdb6c01c349f0d9878405af4cedfaf3505040a31 WatchSource:0}: Error finding container f99ac00ea5b99ab2ec8b39b3bdb6c01c349f0d9878405af4cedfaf3505040a31: Status 404 returned error can't find the container with id f99ac00ea5b99ab2ec8b39b3bdb6c01c349f0d9878405af4cedfaf3505040a31 Jul 30 15:12:04.003745 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.003712 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f875d6977-b47mk"] Jul 30 15:12:04.004115 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:12:04.004062 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-f875d6977-b47mk" podUID="406028a3-cf3c-44ee-83da-39aae2661e87" Jul 30 15:12:04.301649 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.301604 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-76889c7fb4-wm6zs"] Jul 30 15:12:04.304217 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.304195 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.307310 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.307285 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-x2b54\"" Jul 30 15:12:04.308258 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.308064 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 30 15:12:04.308258 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.308111 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 30 15:12:04.311822 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.309032 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 30 15:12:04.311822 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.309838 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 30 15:12:04.311822 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.309894 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-425jnc0h327st\"" Jul 30 15:12:04.319219 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.319196 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76889c7fb4-wm6zs"] Jul 30 15:12:04.344217 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.344186 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/80bbd3b1-ac7f-4992-8196-766b644df1a6-client-ca-bundle\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.344739 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.344711 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/80bbd3b1-ac7f-4992-8196-766b644df1a6-metrics-server-audit-profiles\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.344896 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.344783 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcd84\" (UniqueName: \"kubernetes.io/projected/80bbd3b1-ac7f-4992-8196-766b644df1a6-kube-api-access-dcd84\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.344896 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.344833 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80bbd3b1-ac7f-4992-8196-766b644df1a6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.344896 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.344865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/80bbd3b1-ac7f-4992-8196-766b644df1a6-secret-metrics-server-client-certs\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.345074 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.344920 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/80bbd3b1-ac7f-4992-8196-766b644df1a6-audit-log\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.345074 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.344996 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/80bbd3b1-ac7f-4992-8196-766b644df1a6-secret-metrics-server-tls\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.446210 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.446167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/80bbd3b1-ac7f-4992-8196-766b644df1a6-metrics-server-audit-profiles\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.446390 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.446235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcd84\" (UniqueName: \"kubernetes.io/projected/80bbd3b1-ac7f-4992-8196-766b644df1a6-kube-api-access-dcd84\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.446390 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.446267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80bbd3b1-ac7f-4992-8196-766b644df1a6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.446390 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.446299 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/80bbd3b1-ac7f-4992-8196-766b644df1a6-secret-metrics-server-client-certs\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.446390 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.446324 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/80bbd3b1-ac7f-4992-8196-766b644df1a6-audit-log\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.446608 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.446387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/80bbd3b1-ac7f-4992-8196-766b644df1a6-secret-metrics-server-tls\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.446608 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.446464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/80bbd3b1-ac7f-4992-8196-766b644df1a6-client-ca-bundle\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.447247 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.447222 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/80bbd3b1-ac7f-4992-8196-766b644df1a6-audit-log\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.447456 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.447433 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/80bbd3b1-ac7f-4992-8196-766b644df1a6-metrics-server-audit-profiles\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.447643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.447620 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80bbd3b1-ac7f-4992-8196-766b644df1a6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.449671 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.449646 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/80bbd3b1-ac7f-4992-8196-766b644df1a6-secret-metrics-server-tls\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.449849 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.449788 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/80bbd3b1-ac7f-4992-8196-766b644df1a6-client-ca-bundle\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.451727 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.451676 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/80bbd3b1-ac7f-4992-8196-766b644df1a6-secret-metrics-server-client-certs\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.462990 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.462959 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcd84\" (UniqueName: \"kubernetes.io/projected/80bbd3b1-ac7f-4992-8196-766b644df1a6-kube-api-access-dcd84\") pod \"metrics-server-76889c7fb4-wm6zs\" (UID: \"80bbd3b1-ac7f-4992-8196-766b644df1a6\") " pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.618090 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.617757 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:04.722402 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.722114 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5clt4" event={"ID":"e653e12c-86c8-4cca-897f-a7677c143087","Type":"ContainerStarted","Data":"2e56f421da9f4d1fcbb335524b7649651980b24645b6582badf478363768c85e"} Jul 30 15:12:04.722402 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.722177 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5clt4" event={"ID":"e653e12c-86c8-4cca-897f-a7677c143087","Type":"ContainerStarted","Data":"284b1c654f6e66cf15e50964ecfb4dab7fe2e983846f2944084f5404466ece0d"} Jul 30 15:12:04.724557 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.724494 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8tgkr" event={"ID":"453cbb90-b8f9-48e1-981d-22a44831433d","Type":"ContainerStarted","Data":"81c83ae2fbb6ef3d13b2042aef07df2f810e10a32cb20f9b61f335cb0e4c0b7c"} Jul 30 15:12:04.727182 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.726959 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" event={"ID":"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9","Type":"ContainerStarted","Data":"81db0afa95cde5cbb30e00ed9051c2c0e2c2caa32289dc11363cef8a0db627e5"} Jul 30 15:12:04.727182 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.727019 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" event={"ID":"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9","Type":"ContainerStarted","Data":"7b34835bb8de082a736248becab57d5fca529c22dc1c31aa54409c51a084c92d"} Jul 30 15:12:04.727182 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.727053 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" event={"ID":"ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9","Type":"ContainerStarted","Data":"dba286c67590bad9e9acad3942fa10ab9456ee7392ab16fb8c71290b595fedc8"} Jul 30 15:12:04.729415 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.728407 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:12:04.729415 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.728675 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a","Type":"ContainerStarted","Data":"f99ac00ea5b99ab2ec8b39b3bdb6c01c349f0d9878405af4cedfaf3505040a31"} Jul 30 15:12:04.736447 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.736404 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:12:04.752141 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.752093 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5clt4" podStartSLOduration=4.591181067 podStartE2EDuration="5.752073739s" totalImagesPullingTime="1.077316132s" totalInitContainerRuntime="83.57654ms" isStatefulPod=true podCreationTimestamp="2026-07-30 15:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:12:04.751758156 +0000 UTC m=+151.986889002" watchObservedRunningTime="2026-07-30 15:12:04.752073739 +0000 UTC m=+151.987204585" Jul 30 15:12:04.785056 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.785023 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76889c7fb4-wm6zs"] Jul 30 15:12:04.787989 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.787919 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pdxgg" podStartSLOduration=2.138670198 podStartE2EDuration="5.787901894s" totalImagesPullingTime="3.649231696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:12:04.787828103 +0000 UTC m=+152.022958953" watchObservedRunningTime="2026-07-30 15:12:04.787901894 +0000 UTC m=+152.023032753" Jul 30 15:12:04.849528 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.849498 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/406028a3-cf3c-44ee-83da-39aae2661e87-registry-certificates\") pod \"406028a3-cf3c-44ee-83da-39aae2661e87\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " Jul 30 15:12:04.849698 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.849596 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-bound-sa-token\") pod \"406028a3-cf3c-44ee-83da-39aae2661e87\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " Jul 30 15:12:04.849698 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.849652 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/406028a3-cf3c-44ee-83da-39aae2661e87-trusted-ca\") pod \"406028a3-cf3c-44ee-83da-39aae2661e87\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " Jul 30 15:12:04.849830 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.849699 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlbmv\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-kube-api-access-mlbmv\") pod \"406028a3-cf3c-44ee-83da-39aae2661e87\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " Jul 30 15:12:04.849830 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.849732 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/406028a3-cf3c-44ee-83da-39aae2661e87-ca-trust-extracted\") pod \"406028a3-cf3c-44ee-83da-39aae2661e87\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " Jul 30 15:12:04.849830 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.849768 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/406028a3-cf3c-44ee-83da-39aae2661e87-installation-pull-secrets\") pod \"406028a3-cf3c-44ee-83da-39aae2661e87\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " Jul 30 15:12:04.849830 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.849820 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/406028a3-cf3c-44ee-83da-39aae2661e87-image-registry-private-configuration\") pod \"406028a3-cf3c-44ee-83da-39aae2661e87\" (UID: \"406028a3-cf3c-44ee-83da-39aae2661e87\") " Jul 30 15:12:04.850641 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.850117 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/406028a3-cf3c-44ee-83da-39aae2661e87-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "406028a3-cf3c-44ee-83da-39aae2661e87" (UID: "406028a3-cf3c-44ee-83da-39aae2661e87"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:12:04.850641 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.850253 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/406028a3-cf3c-44ee-83da-39aae2661e87-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "406028a3-cf3c-44ee-83da-39aae2661e87" (UID: "406028a3-cf3c-44ee-83da-39aae2661e87"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:12:04.850641 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.850383 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/406028a3-cf3c-44ee-83da-39aae2661e87-trusted-ca\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:12:04.850641 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.850404 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/406028a3-cf3c-44ee-83da-39aae2661e87-registry-certificates\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:12:04.850958 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.850726 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406028a3-cf3c-44ee-83da-39aae2661e87-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "406028a3-cf3c-44ee-83da-39aae2661e87" (UID: "406028a3-cf3c-44ee-83da-39aae2661e87"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:12:04.853298 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.853242 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/406028a3-cf3c-44ee-83da-39aae2661e87-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "406028a3-cf3c-44ee-83da-39aae2661e87" (UID: "406028a3-cf3c-44ee-83da-39aae2661e87"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:12:04.853298 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.853234 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-kube-api-access-mlbmv" (OuterVolumeSpecName: "kube-api-access-mlbmv") pod "406028a3-cf3c-44ee-83da-39aae2661e87" (UID: "406028a3-cf3c-44ee-83da-39aae2661e87"). InnerVolumeSpecName "kube-api-access-mlbmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:12:04.853592 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.853567 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "406028a3-cf3c-44ee-83da-39aae2661e87" (UID: "406028a3-cf3c-44ee-83da-39aae2661e87"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:12:04.854285 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.854254 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/406028a3-cf3c-44ee-83da-39aae2661e87-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "406028a3-cf3c-44ee-83da-39aae2661e87" (UID: "406028a3-cf3c-44ee-83da-39aae2661e87"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:12:04.951791 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.951701 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlbmv\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-kube-api-access-mlbmv\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:12:04.951791 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.951753 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/406028a3-cf3c-44ee-83da-39aae2661e87-ca-trust-extracted\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:12:04.951791 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.951769 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/406028a3-cf3c-44ee-83da-39aae2661e87-installation-pull-secrets\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:12:04.951791 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.951785 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/406028a3-cf3c-44ee-83da-39aae2661e87-image-registry-private-configuration\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:12:04.951791 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:04.951813 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-bound-sa-token\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:12:04.961629 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:12:04.961593 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80bbd3b1_ac7f_4992_8196_766b644df1a6.slice/crio-6cd3bc53219862c4d5ca11e646f6c6f82440683955ba5131b388f0f9d00e3c8c WatchSource:0}: Error finding container 6cd3bc53219862c4d5ca11e646f6c6f82440683955ba5131b388f0f9d00e3c8c: Status 404 returned error can't find the container with id 6cd3bc53219862c4d5ca11e646f6c6f82440683955ba5131b388f0f9d00e3c8c Jul 30 15:12:05.736888 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.736408 2579 generic.go:356] "Generic (PLEG): container finished" podID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerID="4c27e5bd3adeabf14f1f3e0cca03fa073aeb43f0f69fe88784b25f8df58e93e3" exitCode=0 Jul 30 15:12:05.736888 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.736556 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a","Type":"ContainerDied","Data":"4c27e5bd3adeabf14f1f3e0cca03fa073aeb43f0f69fe88784b25f8df58e93e3"} Jul 30 15:12:05.738782 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.738706 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" event={"ID":"80bbd3b1-ac7f-4992-8196-766b644df1a6","Type":"ContainerStarted","Data":"6cd3bc53219862c4d5ca11e646f6c6f82440683955ba5131b388f0f9d00e3c8c"} Jul 30 15:12:05.739129 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.739106 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f875d6977-b47mk" Jul 30 15:12:05.825283 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.825247 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 15:12:05.830622 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.830595 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.835709 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.835553 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f875d6977-b47mk"] Jul 30 15:12:05.835709 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.835597 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 30 15:12:05.835972 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.835877 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 30 15:12:05.846209 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.846102 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 30 15:12:05.846209 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.846133 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 30 15:12:05.846209 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.846170 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 30 15:12:05.847958 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.847785 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 30 15:12:05.847958 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.847812 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 30 15:12:05.847958 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.847785 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 30 15:12:05.848171 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.847995 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 30 15:12:05.848339 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.848312 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 30 15:12:05.848449 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.848418 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 30 15:12:05.848586 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.848467 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 30 15:12:05.848586 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.848504 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8dn27m6fkh44t\"" Jul 30 15:12:05.848712 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.848701 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5zfp2\"" Jul 30 15:12:05.851046 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.851023 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-f875d6977-b47mk"] Jul 30 15:12:05.857835 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.857139 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 30 15:12:05.864354 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.864166 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 15:12:05.869431 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.869406 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 30 15:12:05.966104 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966066 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.966104 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966116 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc0f4712-e8fc-40c0-9e02-96996498bf5e-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.966340 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966193 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.966340 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966237 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.966340 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.966340 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966305 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.966593 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966355 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.966593 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966390 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.966593 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966421 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.966593 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966452 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.966593 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966504 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.966593 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966537 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-config\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.966593 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.969355 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966623 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.969355 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966651 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.969355 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966779 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.969355 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.969355 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966890 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc0f4712-e8fc-40c0-9e02-96996498bf5e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.969355 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.966912 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcmp8\" (UniqueName: \"kubernetes.io/projected/fc0f4712-e8fc-40c0-9e02-96996498bf5e-kube-api-access-dcmp8\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.969355 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.969324 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:05.969707 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:05.969435 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/406028a3-cf3c-44ee-83da-39aae2661e87-registry-tls\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:12:06.070156 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.070076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.070467 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.070161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.070467 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.070192 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc0f4712-e8fc-40c0-9e02-96996498bf5e-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.070467 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.070222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.070467 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.070247 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.070467 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.070262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.070467 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.070289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.075869 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.071274 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.075869 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.071567 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.075869 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.072067 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.075869 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.072225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.075869 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.072257 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.075869 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.072292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.075869 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.072326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.075869 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.072348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.075869 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.072384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-config\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.075869 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.072408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.075869 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.072436 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.075869 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.072459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.075869 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.072503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.075869 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.072529 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.075869 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.072565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc0f4712-e8fc-40c0-9e02-96996498bf5e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.075869 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.072589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcmp8\" (UniqueName: \"kubernetes.io/projected/fc0f4712-e8fc-40c0-9e02-96996498bf5e-kube-api-access-dcmp8\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.075869 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.073232 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.076698 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.073465 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.076698 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.073882 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.076698 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.074093 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.076698 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.074127 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.076698 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.074830 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.078895 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.078852 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.079301 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.079242 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc0f4712-e8fc-40c0-9e02-96996498bf5e-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.079643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.079599 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.080620 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.080593 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-config\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.081407 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.081387 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.082143 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.082120 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.082670 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.082613 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc0f4712-e8fc-40c0-9e02-96996498bf5e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.082822 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.082766 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.083311 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.083284 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.084544 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.084520 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.086788 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.086749 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcmp8\" (UniqueName: \"kubernetes.io/projected/fc0f4712-e8fc-40c0-9e02-96996498bf5e-kube-api-access-dcmp8\") pod \"prometheus-k8s-0\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:06.144960 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:06.144924 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:07.197536 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:07.197497 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="406028a3-cf3c-44ee-83da-39aae2661e87" path="/var/lib/kubelet/pods/406028a3-cf3c-44ee-83da-39aae2661e87/volumes" Jul 30 15:12:08.315305 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:08.315277 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 15:12:08.317124 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:12:08.317094 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0f4712_e8fc_40c0_9e02_96996498bf5e.slice/crio-7a03f80a485f670be569e628c39987ef1cf902df86bad9c7567e18bd872e5b3d WatchSource:0}: Error finding container 7a03f80a485f670be569e628c39987ef1cf902df86bad9c7567e18bd872e5b3d: Status 404 returned error can't find the container with id 7a03f80a485f670be569e628c39987ef1cf902df86bad9c7567e18bd872e5b3d Jul 30 15:12:08.757372 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:08.757262 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" event={"ID":"80bbd3b1-ac7f-4992-8196-766b644df1a6","Type":"ContainerStarted","Data":"c36279b189668e2bb694d14548b879ed2a0d356a4e0139601942ba7a2aa6c86c"} Jul 30 15:12:08.759342 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:08.759307 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8tgkr" event={"ID":"453cbb90-b8f9-48e1-981d-22a44831433d","Type":"ContainerStarted","Data":"2343fe9aa35e9965551793424e0d9357cef547e60058903768223ea0380f8773"} Jul 30 15:12:08.760783 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:08.760756 2579 generic.go:356] "Generic (PLEG): container finished" podID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerID="95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45" exitCode=0 Jul 30 15:12:08.760911 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:08.760810 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc0f4712-e8fc-40c0-9e02-96996498bf5e","Type":"ContainerDied","Data":"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45"} Jul 30 15:12:08.760911 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:08.760867 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc0f4712-e8fc-40c0-9e02-96996498bf5e","Type":"ContainerStarted","Data":"7a03f80a485f670be569e628c39987ef1cf902df86bad9c7567e18bd872e5b3d"} Jul 30 15:12:08.800342 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:08.800295 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" podStartSLOduration=1.624934975 podStartE2EDuration="4.800280319s" totalImagesPullingTime="3.175345344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:12:08.799073194 +0000 UTC m=+156.034204044" watchObservedRunningTime="2026-07-30 15:12:08.800280319 +0000 UTC m=+156.035411165" Jul 30 15:12:08.837598 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:08.837544 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8tgkr" podStartSLOduration=65.729735456 podStartE2EDuration="1m18.837524348s" totalImagesPullingTime="13.107788892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 15:10:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:12:08.836733144 +0000 UTC m=+156.071863992" watchObservedRunningTime="2026-07-30 15:12:08.837524348 +0000 UTC m=+156.072655203" Jul 30 15:12:09.768402 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:09.768312 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a","Type":"ContainerStarted","Data":"516975b1377857fd9040e25d4543c6fc428f6681033500d8a62328e190d2825e"} Jul 30 15:12:09.768737 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:09.768428 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a","Type":"ContainerStarted","Data":"ecfc0b84131730c95b1897bbd73f077cc4eb6d37fac4cb9a702c271a6da4a1dd"} Jul 30 15:12:09.768737 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:09.768459 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a","Type":"ContainerStarted","Data":"fa3528257cafd695436f4de0e6712d14c6de562d107e69460c45c3e802145c61"} Jul 30 15:12:09.768737 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:09.768493 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a","Type":"ContainerStarted","Data":"e7d2044eb41f7832964f1e67ee7be9e05da0e952c54c632976b85e653c5a16c1"} Jul 30 15:12:09.768737 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:09.768520 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a","Type":"ContainerStarted","Data":"3cb7958dd3cdb4266b427425d33bc9916b658cc294d0c450ecc6d3c168f9bf1c"} Jul 30 15:12:10.779036 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:10.779000 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a","Type":"ContainerStarted","Data":"0e0435dc9c24926f6f4d087e90741c03c67b140a7081625a8b3c339aa51f2092"} Jul 30 15:12:11.148473 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:12:11.148424 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gq5ph" podUID="829b9cdd-657c-4f62-8a83-813c43eefa09" Jul 30 15:12:11.783217 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:11.783178 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gq5ph" Jul 30 15:12:13.793400 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:13.793354 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc0f4712-e8fc-40c0-9e02-96996498bf5e","Type":"ContainerStarted","Data":"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb"} Jul 30 15:12:13.793837 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:13.793456 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc0f4712-e8fc-40c0-9e02-96996498bf5e","Type":"ContainerStarted","Data":"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa"} Jul 30 15:12:13.794967 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:13.794937 2579 generic.go:356] "Generic (PLEG): container finished" podID="0cc61754-0414-4c86-84de-250ba68dc560" containerID="09ad6c81a612c465aef682392e3cce3c0398ec4881c5cda54684dc19d0ff3116" exitCode=0 Jul 30 15:12:13.795075 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:13.794997 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q" event={"ID":"0cc61754-0414-4c86-84de-250ba68dc560","Type":"ContainerDied","Data":"09ad6c81a612c465aef682392e3cce3c0398ec4881c5cda54684dc19d0ff3116"} Jul 30 15:12:13.795370 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:13.795349 2579 scope.go:117] "RemoveContainer" containerID="09ad6c81a612c465aef682392e3cce3c0398ec4881c5cda54684dc19d0ff3116" Jul 30 15:12:13.820652 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:13.820611 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=8.093528607 podStartE2EDuration="13.820596036s" totalImagesPullingTime="5.70737911s" totalInitContainerRuntime="19.688319ms" isStatefulPod=false podCreationTimestamp="2026-07-30 15:12:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:12:10.829058822 +0000 UTC m=+158.064189704" watchObservedRunningTime="2026-07-30 15:12:13.820596036 +0000 UTC m=+161.055726881" Jul 30 15:12:14.801908 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:14.801864 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q" event={"ID":"0cc61754-0414-4c86-84de-250ba68dc560","Type":"ContainerStarted","Data":"114eb481e543c00ad4ad95a87496e36ac0c545a4468d1e02c2ad3cac638ccce6"} Jul 30 15:12:15.808405 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:15.808372 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc0f4712-e8fc-40c0-9e02-96996498bf5e","Type":"ContainerStarted","Data":"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0"} Jul 30 15:12:15.808775 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:15.808464 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc0f4712-e8fc-40c0-9e02-96996498bf5e","Type":"ContainerStarted","Data":"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57"} Jul 30 15:12:15.808775 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:15.808493 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc0f4712-e8fc-40c0-9e02-96996498bf5e","Type":"ContainerStarted","Data":"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36"} Jul 30 15:12:15.808775 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:15.808516 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc0f4712-e8fc-40c0-9e02-96996498bf5e","Type":"ContainerStarted","Data":"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12"} Jul 30 15:12:16.078126 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:16.078049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:12:16.080252 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:16.080223 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/829b9cdd-657c-4f62-8a83-813c43eefa09-metrics-tls\") pod \"dns-default-gq5ph\" (UID: \"829b9cdd-657c-4f62-8a83-813c43eefa09\") " pod="openshift-dns/dns-default-gq5ph" Jul 30 15:12:16.145452 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:16.145401 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:12:16.287618 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:16.287585 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7hzbg\"" Jul 30 15:12:16.295217 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:16.295199 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gq5ph" Jul 30 15:12:16.427196 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:16.427132 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.251398931 podStartE2EDuration="11.427106707s" totalImagesPullingTime="6.158546737s" totalInitContainerRuntime="17.161039ms" isStatefulPod=false podCreationTimestamp="2026-07-30 15:12:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:12:15.879132367 +0000 UTC m=+163.114263212" watchObservedRunningTime="2026-07-30 15:12:16.427106707 +0000 UTC m=+163.662237556" Jul 30 15:12:16.428743 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:16.428720 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gq5ph"] Jul 30 15:12:16.430680 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:12:16.430654 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829b9cdd_657c_4f62_8a83_813c43eefa09.slice/crio-6824d0b5fbc9a196053aa27d10ce92ec5f3335f2f5172e867d7c5f1bea0b630f WatchSource:0}: Error finding container 6824d0b5fbc9a196053aa27d10ce92ec5f3335f2f5172e867d7c5f1bea0b630f: Status 404 returned error can't find the container with id 6824d0b5fbc9a196053aa27d10ce92ec5f3335f2f5172e867d7c5f1bea0b630f Jul 30 15:12:16.813450 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:16.813406 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gq5ph" event={"ID":"829b9cdd-657c-4f62-8a83-813c43eefa09","Type":"ContainerStarted","Data":"6824d0b5fbc9a196053aa27d10ce92ec5f3335f2f5172e867d7c5f1bea0b630f"} Jul 30 15:12:18.978092 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:18.978049 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cvbfc_c35ba36d-8199-471f-99df-70cc5ba0c05e/dns-node-resolver/0.log" Jul 30 15:12:19.573997 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:19.573967 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c8b769f9f-tfnst_04216c4f-4a2e-4c3b-a254-b2c11ad44244/router/0.log" Jul 30 15:12:19.804723 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:19.804697 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ksbjt_d5098562-bcb2-4825-8429-184d57da5659/serve-healthcheck-canary/0.log" Jul 30 15:12:19.828540 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:19.828479 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gq5ph" event={"ID":"829b9cdd-657c-4f62-8a83-813c43eefa09","Type":"ContainerStarted","Data":"8bfc112bf39e0da12caa23e29a8836b3c6fecbe2a69644d4498b1be9d949e202"} Jul 30 15:12:19.828540 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:19.828526 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gq5ph" event={"ID":"829b9cdd-657c-4f62-8a83-813c43eefa09","Type":"ContainerStarted","Data":"3287293737b24bad768eb6f4b45b2de8fba64d6d687114a95fea190d19e11ce3"} Jul 30 15:12:19.828663 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:19.828579 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gq5ph" Jul 30 15:12:24.618704 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:24.618672 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:24.619074 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:24.618733 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:29.835107 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:29.835029 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gq5ph" Jul 30 15:12:29.863451 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:29.863396 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gq5ph" podStartSLOduration=139.367183006 podStartE2EDuration="2m21.86337602s" totalImagesPullingTime="2.496193014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:12:19.874884562 +0000 UTC m=+167.110015407" watchObservedRunningTime="2026-07-30 15:12:29.86337602 +0000 UTC m=+177.098506873" Jul 30 15:12:44.623827 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:44.623777 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:12:44.627621 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:12:44.627601 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-76889c7fb4-wm6zs" Jul 30 15:13:06.145097 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:06.145055 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:06.160151 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:06.160130 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:07.029276 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:07.029250 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:19.979277 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:19.979241 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 15:13:19.979722 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:19.979547 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="alertmanager" containerID="cri-o://3cb7958dd3cdb4266b427425d33bc9916b658cc294d0c450ecc6d3c168f9bf1c" gracePeriod=120 Jul 30 15:13:19.979722 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:19.979556 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="prom-label-proxy" containerID="cri-o://0e0435dc9c24926f6f4d087e90741c03c67b140a7081625a8b3c339aa51f2092" gracePeriod=120 Jul 30 15:13:19.979722 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:19.979579 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="config-reloader" containerID="cri-o://e7d2044eb41f7832964f1e67ee7be9e05da0e952c54c632976b85e653c5a16c1" gracePeriod=120 Jul 30 15:13:19.979722 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:19.979559 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="kube-rbac-proxy-web" containerID="cri-o://fa3528257cafd695436f4de0e6712d14c6de562d107e69460c45c3e802145c61" gracePeriod=120 Jul 30 15:13:19.979722 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:19.979611 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="kube-rbac-proxy-metric" containerID="cri-o://516975b1377857fd9040e25d4543c6fc428f6681033500d8a62328e190d2825e" gracePeriod=120 Jul 30 15:13:19.979722 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:19.979618 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="kube-rbac-proxy" containerID="cri-o://ecfc0b84131730c95b1897bbd73f077cc4eb6d37fac4cb9a702c271a6da4a1dd" gracePeriod=120 Jul 30 15:13:21.069841 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.069812 2579 generic.go:356] "Generic (PLEG): container finished" podID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerID="0e0435dc9c24926f6f4d087e90741c03c67b140a7081625a8b3c339aa51f2092" exitCode=0 Jul 30 15:13:21.069841 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.069826 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a","Type":"ContainerDied","Data":"0e0435dc9c24926f6f4d087e90741c03c67b140a7081625a8b3c339aa51f2092"} Jul 30 15:13:21.070179 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.069895 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a","Type":"ContainerDied","Data":"516975b1377857fd9040e25d4543c6fc428f6681033500d8a62328e190d2825e"} Jul 30 15:13:21.070179 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.069838 2579 generic.go:356] "Generic (PLEG): container finished" podID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerID="516975b1377857fd9040e25d4543c6fc428f6681033500d8a62328e190d2825e" exitCode=0 Jul 30 15:13:21.070179 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.069922 2579 generic.go:356] "Generic (PLEG): container finished" podID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerID="ecfc0b84131730c95b1897bbd73f077cc4eb6d37fac4cb9a702c271a6da4a1dd" exitCode=0 Jul 30 15:13:21.070179 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.069930 2579 generic.go:356] "Generic (PLEG): container finished" podID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerID="e7d2044eb41f7832964f1e67ee7be9e05da0e952c54c632976b85e653c5a16c1" exitCode=0 Jul 30 15:13:21.070179 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.069936 2579 generic.go:356] "Generic (PLEG): container finished" podID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerID="3cb7958dd3cdb4266b427425d33bc9916b658cc294d0c450ecc6d3c168f9bf1c" exitCode=0 Jul 30 15:13:21.070179 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.069983 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a","Type":"ContainerDied","Data":"ecfc0b84131730c95b1897bbd73f077cc4eb6d37fac4cb9a702c271a6da4a1dd"} Jul 30 15:13:21.070179 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.070026 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a","Type":"ContainerDied","Data":"e7d2044eb41f7832964f1e67ee7be9e05da0e952c54c632976b85e653c5a16c1"} Jul 30 15:13:21.070179 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.070051 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a","Type":"ContainerDied","Data":"3cb7958dd3cdb4266b427425d33bc9916b658cc294d0c450ecc6d3c168f9bf1c"} Jul 30 15:13:21.216228 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.216207 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:21.397018 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.396994 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-kube-rbac-proxy-web\") pod \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " Jul 30 15:13:21.397182 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.397038 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-alertmanager-main-db\") pod \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " Jul 30 15:13:21.397182 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.397055 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-cluster-tls-config\") pod \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " Jul 30 15:13:21.397182 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.397109 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76nsr\" (UniqueName: \"kubernetes.io/projected/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-kube-api-access-76nsr\") pod \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " Jul 30 15:13:21.397182 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.397128 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-tls-assets\") pod \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " Jul 30 15:13:21.397403 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.397153 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-alertmanager-trusted-ca-bundle\") pod \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " Jul 30 15:13:21.397403 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.397223 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-kube-rbac-proxy\") pod \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " Jul 30 15:13:21.397403 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.397253 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-config-volume\") pod \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " Jul 30 15:13:21.397403 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.397283 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-config-out\") pod \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " Jul 30 15:13:21.397403 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.397309 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-metrics-client-ca\") pod \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " Jul 30 15:13:21.397403 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.397346 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" (UID: "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:13:21.397403 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.397364 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " Jul 30 15:13:21.397403 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.397402 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-main-tls\") pod \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " Jul 30 15:13:21.397834 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.397428 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-web-config\") pod \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\" (UID: \"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a\") " Jul 30 15:13:21.397834 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.397678 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-alertmanager-main-db\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:21.397834 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.397720 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" (UID: "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:13:21.397834 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.397732 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" (UID: "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:13:21.400493 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.400458 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-kube-api-access-76nsr" (OuterVolumeSpecName: "kube-api-access-76nsr") pod "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" (UID: "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a"). InnerVolumeSpecName "kube-api-access-76nsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:13:21.400879 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.400850 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" (UID: "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:13:21.401011 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.400892 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" (UID: "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:13:21.401011 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.400967 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-config-volume" (OuterVolumeSpecName: "config-volume") pod "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" (UID: "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:13:21.401261 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.401238 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" (UID: "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:13:21.401343 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.401256 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" (UID: "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:13:21.401614 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.401595 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-config-out" (OuterVolumeSpecName: "config-out") pod "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" (UID: "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:13:21.401694 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.401660 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" (UID: "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:13:21.404285 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.404261 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" (UID: "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:13:21.410453 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.410427 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-web-config" (OuterVolumeSpecName: "web-config") pod "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" (UID: "8fe47b82-3ab6-4505-99bf-b6c5bb503a0a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:13:21.498613 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.498591 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76nsr\" (UniqueName: \"kubernetes.io/projected/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-kube-api-access-76nsr\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:21.498697 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.498617 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-tls-assets\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:21.498697 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.498627 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:21.498697 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.498637 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:21.498697 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.498646 2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:21.498697 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.498654 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-config-out\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:21.498697 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.498661 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-metrics-client-ca\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:21.498697 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.498669 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:21.498697 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.498679 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-main-tls\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:21.498697 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.498693 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-web-config\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:21.498981 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.498702 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:21.498981 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:21.498710 2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a-cluster-tls-config\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:22.077516 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.077484 2579 generic.go:356] "Generic (PLEG): container finished" podID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerID="fa3528257cafd695436f4de0e6712d14c6de562d107e69460c45c3e802145c61" exitCode=0 Jul 30 15:13:22.077955 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.077568 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.077955 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.077573 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a","Type":"ContainerDied","Data":"fa3528257cafd695436f4de0e6712d14c6de562d107e69460c45c3e802145c61"} Jul 30 15:13:22.077955 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.077700 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fe47b82-3ab6-4505-99bf-b6c5bb503a0a","Type":"ContainerDied","Data":"f99ac00ea5b99ab2ec8b39b3bdb6c01c349f0d9878405af4cedfaf3505040a31"} Jul 30 15:13:22.077955 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.077756 2579 scope.go:117] "RemoveContainer" containerID="0e0435dc9c24926f6f4d087e90741c03c67b140a7081625a8b3c339aa51f2092" Jul 30 15:13:22.085138 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.085120 2579 scope.go:117] "RemoveContainer" containerID="516975b1377857fd9040e25d4543c6fc428f6681033500d8a62328e190d2825e" Jul 30 15:13:22.094138 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.094118 2579 scope.go:117] "RemoveContainer" containerID="ecfc0b84131730c95b1897bbd73f077cc4eb6d37fac4cb9a702c271a6da4a1dd" Jul 30 15:13:22.100821 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.100791 2579 scope.go:117] "RemoveContainer" containerID="fa3528257cafd695436f4de0e6712d14c6de562d107e69460c45c3e802145c61" Jul 30 15:13:22.107636 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.107614 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 15:13:22.107938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.107922 2579 scope.go:117] "RemoveContainer" containerID="e7d2044eb41f7832964f1e67ee7be9e05da0e952c54c632976b85e653c5a16c1" Jul 30 15:13:22.114420 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.114404 2579 scope.go:117] "RemoveContainer" containerID="3cb7958dd3cdb4266b427425d33bc9916b658cc294d0c450ecc6d3c168f9bf1c" Jul 30 15:13:22.116882 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.116861 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 15:13:22.121329 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.121314 2579 scope.go:117] "RemoveContainer" containerID="4c27e5bd3adeabf14f1f3e0cca03fa073aeb43f0f69fe88784b25f8df58e93e3" Jul 30 15:13:22.127945 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.127919 2579 scope.go:117] "RemoveContainer" containerID="0e0435dc9c24926f6f4d087e90741c03c67b140a7081625a8b3c339aa51f2092" Jul 30 15:13:22.128172 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:13:22.128155 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e0435dc9c24926f6f4d087e90741c03c67b140a7081625a8b3c339aa51f2092\": container with ID starting with 0e0435dc9c24926f6f4d087e90741c03c67b140a7081625a8b3c339aa51f2092 not found: ID does not exist" containerID="0e0435dc9c24926f6f4d087e90741c03c67b140a7081625a8b3c339aa51f2092" Jul 30 15:13:22.128232 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.128179 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e0435dc9c24926f6f4d087e90741c03c67b140a7081625a8b3c339aa51f2092"} err="failed to get container status \"0e0435dc9c24926f6f4d087e90741c03c67b140a7081625a8b3c339aa51f2092\": rpc error: code = NotFound desc = could not find container \"0e0435dc9c24926f6f4d087e90741c03c67b140a7081625a8b3c339aa51f2092\": container with ID starting with 0e0435dc9c24926f6f4d087e90741c03c67b140a7081625a8b3c339aa51f2092 not found: ID does not exist" Jul 30 15:13:22.128232 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.128207 2579 scope.go:117] "RemoveContainer" containerID="516975b1377857fd9040e25d4543c6fc428f6681033500d8a62328e190d2825e" Jul 30 15:13:22.128413 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:13:22.128395 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516975b1377857fd9040e25d4543c6fc428f6681033500d8a62328e190d2825e\": container with ID starting with 516975b1377857fd9040e25d4543c6fc428f6681033500d8a62328e190d2825e not found: ID does not exist" containerID="516975b1377857fd9040e25d4543c6fc428f6681033500d8a62328e190d2825e" Jul 30 15:13:22.128474 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.128422 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516975b1377857fd9040e25d4543c6fc428f6681033500d8a62328e190d2825e"} err="failed to get container status \"516975b1377857fd9040e25d4543c6fc428f6681033500d8a62328e190d2825e\": rpc error: code = NotFound desc = could not find container \"516975b1377857fd9040e25d4543c6fc428f6681033500d8a62328e190d2825e\": container with ID starting with 516975b1377857fd9040e25d4543c6fc428f6681033500d8a62328e190d2825e not found: ID does not exist" Jul 30 15:13:22.128474 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.128446 2579 scope.go:117] "RemoveContainer" containerID="ecfc0b84131730c95b1897bbd73f077cc4eb6d37fac4cb9a702c271a6da4a1dd" Jul 30 15:13:22.128703 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:13:22.128687 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecfc0b84131730c95b1897bbd73f077cc4eb6d37fac4cb9a702c271a6da4a1dd\": container with ID starting with ecfc0b84131730c95b1897bbd73f077cc4eb6d37fac4cb9a702c271a6da4a1dd not found: ID does not exist" containerID="ecfc0b84131730c95b1897bbd73f077cc4eb6d37fac4cb9a702c271a6da4a1dd" Jul 30 15:13:22.128745 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.128708 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecfc0b84131730c95b1897bbd73f077cc4eb6d37fac4cb9a702c271a6da4a1dd"} err="failed to get container status \"ecfc0b84131730c95b1897bbd73f077cc4eb6d37fac4cb9a702c271a6da4a1dd\": rpc error: code = NotFound desc = could not find container \"ecfc0b84131730c95b1897bbd73f077cc4eb6d37fac4cb9a702c271a6da4a1dd\": container with ID starting with ecfc0b84131730c95b1897bbd73f077cc4eb6d37fac4cb9a702c271a6da4a1dd not found: ID does not exist" Jul 30 15:13:22.128745 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.128724 2579 scope.go:117] "RemoveContainer" containerID="fa3528257cafd695436f4de0e6712d14c6de562d107e69460c45c3e802145c61" Jul 30 15:13:22.128936 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:13:22.128920 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3528257cafd695436f4de0e6712d14c6de562d107e69460c45c3e802145c61\": container with ID starting with fa3528257cafd695436f4de0e6712d14c6de562d107e69460c45c3e802145c61 not found: ID does not exist" containerID="fa3528257cafd695436f4de0e6712d14c6de562d107e69460c45c3e802145c61" Jul 30 15:13:22.128990 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.128940 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3528257cafd695436f4de0e6712d14c6de562d107e69460c45c3e802145c61"} err="failed to get container status \"fa3528257cafd695436f4de0e6712d14c6de562d107e69460c45c3e802145c61\": rpc error: code = NotFound desc = could not find container \"fa3528257cafd695436f4de0e6712d14c6de562d107e69460c45c3e802145c61\": container with ID starting with fa3528257cafd695436f4de0e6712d14c6de562d107e69460c45c3e802145c61 not found: ID does not exist" Jul 30 15:13:22.128990 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.128953 2579 scope.go:117] "RemoveContainer" containerID="e7d2044eb41f7832964f1e67ee7be9e05da0e952c54c632976b85e653c5a16c1" Jul 30 15:13:22.129176 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:13:22.129160 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d2044eb41f7832964f1e67ee7be9e05da0e952c54c632976b85e653c5a16c1\": container with ID starting with e7d2044eb41f7832964f1e67ee7be9e05da0e952c54c632976b85e653c5a16c1 not found: ID does not exist" containerID="e7d2044eb41f7832964f1e67ee7be9e05da0e952c54c632976b85e653c5a16c1" Jul 30 15:13:22.129215 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.129179 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d2044eb41f7832964f1e67ee7be9e05da0e952c54c632976b85e653c5a16c1"} err="failed to get container status \"e7d2044eb41f7832964f1e67ee7be9e05da0e952c54c632976b85e653c5a16c1\": rpc error: code = NotFound desc = could not find container \"e7d2044eb41f7832964f1e67ee7be9e05da0e952c54c632976b85e653c5a16c1\": container with ID starting with e7d2044eb41f7832964f1e67ee7be9e05da0e952c54c632976b85e653c5a16c1 not found: ID does not exist" Jul 30 15:13:22.129215 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.129191 2579 scope.go:117] "RemoveContainer" containerID="3cb7958dd3cdb4266b427425d33bc9916b658cc294d0c450ecc6d3c168f9bf1c" Jul 30 15:13:22.129399 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:13:22.129380 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb7958dd3cdb4266b427425d33bc9916b658cc294d0c450ecc6d3c168f9bf1c\": container with ID starting with 3cb7958dd3cdb4266b427425d33bc9916b658cc294d0c450ecc6d3c168f9bf1c not found: ID does not exist" containerID="3cb7958dd3cdb4266b427425d33bc9916b658cc294d0c450ecc6d3c168f9bf1c" Jul 30 15:13:22.129473 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.129405 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb7958dd3cdb4266b427425d33bc9916b658cc294d0c450ecc6d3c168f9bf1c"} err="failed to get container status \"3cb7958dd3cdb4266b427425d33bc9916b658cc294d0c450ecc6d3c168f9bf1c\": rpc error: code = NotFound desc = could not find container \"3cb7958dd3cdb4266b427425d33bc9916b658cc294d0c450ecc6d3c168f9bf1c\": container with ID starting with 3cb7958dd3cdb4266b427425d33bc9916b658cc294d0c450ecc6d3c168f9bf1c not found: ID does not exist" Jul 30 15:13:22.129473 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.129424 2579 scope.go:117] "RemoveContainer" containerID="4c27e5bd3adeabf14f1f3e0cca03fa073aeb43f0f69fe88784b25f8df58e93e3" Jul 30 15:13:22.129650 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:13:22.129634 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c27e5bd3adeabf14f1f3e0cca03fa073aeb43f0f69fe88784b25f8df58e93e3\": container with ID starting with 4c27e5bd3adeabf14f1f3e0cca03fa073aeb43f0f69fe88784b25f8df58e93e3 not found: ID does not exist" containerID="4c27e5bd3adeabf14f1f3e0cca03fa073aeb43f0f69fe88784b25f8df58e93e3" Jul 30 15:13:22.129688 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.129658 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c27e5bd3adeabf14f1f3e0cca03fa073aeb43f0f69fe88784b25f8df58e93e3"} err="failed to get container status \"4c27e5bd3adeabf14f1f3e0cca03fa073aeb43f0f69fe88784b25f8df58e93e3\": rpc error: code = NotFound desc = could not find container \"4c27e5bd3adeabf14f1f3e0cca03fa073aeb43f0f69fe88784b25f8df58e93e3\": container with ID starting with 4c27e5bd3adeabf14f1f3e0cca03fa073aeb43f0f69fe88784b25f8df58e93e3 not found: ID does not exist" Jul 30 15:13:22.166936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.166903 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 15:13:22.167218 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167203 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="config-reloader" Jul 30 15:13:22.167298 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167221 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="config-reloader" Jul 30 15:13:22.167298 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167233 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="kube-rbac-proxy-metric" Jul 30 15:13:22.167298 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167243 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="kube-rbac-proxy-metric" Jul 30 15:13:22.167298 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167254 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="alertmanager" Jul 30 15:13:22.167298 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167263 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="alertmanager" Jul 30 15:13:22.167298 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167280 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="kube-rbac-proxy" Jul 30 15:13:22.167298 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167288 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="kube-rbac-proxy" Jul 30 15:13:22.167677 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167307 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="kube-rbac-proxy-web" Jul 30 15:13:22.167677 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167315 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="kube-rbac-proxy-web" Jul 30 15:13:22.167677 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167337 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="init-config-reloader" Jul 30 15:13:22.167677 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167345 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="init-config-reloader" Jul 30 15:13:22.167677 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167354 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="prom-label-proxy" Jul 30 15:13:22.167677 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167362 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="prom-label-proxy" Jul 30 15:13:22.167677 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167429 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="kube-rbac-proxy-metric" Jul 30 15:13:22.167677 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167442 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="kube-rbac-proxy" Jul 30 15:13:22.167677 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167451 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="kube-rbac-proxy-web" Jul 30 15:13:22.167677 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167461 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="alertmanager" Jul 30 15:13:22.167677 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167471 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="config-reloader" Jul 30 15:13:22.167677 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.167480 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" containerName="prom-label-proxy" Jul 30 15:13:22.171470 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.171451 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.177749 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.177691 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 30 15:13:22.177969 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.177950 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 30 15:13:22.178073 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.178019 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 30 15:13:22.178255 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.178240 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 30 15:13:22.178603 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.178584 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 30 15:13:22.178708 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.178691 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 30 15:13:22.178781 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.178766 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 30 15:13:22.178878 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.178861 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 30 15:13:22.179142 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.179127 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qg6rg\"" Jul 30 15:13:22.187433 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.187416 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 30 15:13:22.211732 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.211707 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 15:13:22.306015 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.305989 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.306125 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.306024 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.306125 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.306052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.306125 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.306069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.306125 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.306107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.306289 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.306143 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.306289 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.306169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-config-out\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.306289 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.306189 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-config-volume\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.306289 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.306211 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fctl7\" (UniqueName: \"kubernetes.io/projected/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-kube-api-access-fctl7\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.306289 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.306240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.306289 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.306262 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.306289 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.306285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-web-config\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.306506 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.306314 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.407452 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.407414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.407577 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.407471 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.407642 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.407580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.407642 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.407607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.407642 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.407632 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.407826 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.407660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-config-out\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.407826 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.407690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-config-volume\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.407826 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.407724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fctl7\" (UniqueName: \"kubernetes.io/projected/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-kube-api-access-fctl7\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.407826 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.407761 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.407826 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.407819 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.408076 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.407864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-web-config\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.408076 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.407894 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.408076 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.407901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.408076 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.407949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.408526 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.408498 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.408646 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.408625 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.410375 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.410315 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.410702 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.410631 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.410837 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.410720 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-config-out\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.411050 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.411017 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.411174 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.411154 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.411216 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.411186 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.411256 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.411221 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-web-config\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.411422 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.411400 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-config-volume\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.412527 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.412512 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.422601 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.422583 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fctl7\" (UniqueName: \"kubernetes.io/projected/599c3a66-9f9e-42fa-8c2e-09c59a20a3ed-kube-api-access-fctl7\") pod \"alertmanager-main-0\" (UID: \"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.480469 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.480446 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 15:13:22.646363 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:22.646337 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 15:13:23.083200 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:23.083125 2579 generic.go:356] "Generic (PLEG): container finished" podID="599c3a66-9f9e-42fa-8c2e-09c59a20a3ed" containerID="29a673ae468cd7344488ae2c09bc05577fa90570f59f7721780b42fbc4113cc3" exitCode=0 Jul 30 15:13:23.083622 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:23.083213 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed","Type":"ContainerDied","Data":"29a673ae468cd7344488ae2c09bc05577fa90570f59f7721780b42fbc4113cc3"} Jul 30 15:13:23.083622 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:23.083300 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed","Type":"ContainerStarted","Data":"f7f08a7b2adf907024ef651eaa2dcf063efbbb710a861aab58015e8b4646643f"} Jul 30 15:13:23.196550 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:23.196520 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fe47b82-3ab6-4505-99bf-b6c5bb503a0a" path="/var/lib/kubelet/pods/8fe47b82-3ab6-4505-99bf-b6c5bb503a0a/volumes" Jul 30 15:13:24.058021 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.057991 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-8466886869-jwkvt"] Jul 30 15:13:24.060663 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.060635 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.064132 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.064113 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 30 15:13:24.065096 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.065074 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 30 15:13:24.065484 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.065464 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-679qk\"" Jul 30 15:13:24.065673 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.065657 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 30 15:13:24.065938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.065916 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 30 15:13:24.066171 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.066155 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 30 15:13:24.071741 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.071437 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 30 15:13:24.092861 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.092825 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8466886869-jwkvt"] Jul 30 15:13:24.096489 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.096462 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed","Type":"ContainerStarted","Data":"540c188df84e24aec74c464c77e828d9fbfc13add08529c0e29ec281084b819d"} Jul 30 15:13:24.096598 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.096518 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed","Type":"ContainerStarted","Data":"cbc10c183bf74c3e67c02b242758f8b0d9444e8af07024ec13cba1433e8c8ce1"} Jul 30 15:13:24.096598 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.096543 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed","Type":"ContainerStarted","Data":"350cf6711399909303df3690082f85b4519b46c195a53c58d2f6878e5bb72499"} Jul 30 15:13:24.096598 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.096567 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed","Type":"ContainerStarted","Data":"ac00aee27148bcb1bdd883524ed984886063cf2704afa7aa2ada3fdac505bd97"} Jul 30 15:13:24.096598 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.096589 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed","Type":"ContainerStarted","Data":"9cace9de9e69cda5eb5bd05c857df9e983340f7e633933492d3426f9e42b9806"} Jul 30 15:13:24.096821 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.096612 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"599c3a66-9f9e-42fa-8c2e-09c59a20a3ed","Type":"ContainerStarted","Data":"f7fb9b8ce6ed60b6d2fc0e4e75c12af910cc8dd6e5a3e9930f02ab0fb229c48b"} Jul 30 15:13:24.155771 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.155724 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.140043752 podStartE2EDuration="2.155710164s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.666412ms" isStatefulPod=false podCreationTimestamp="2026-07-30 15:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:13:24.154765925 +0000 UTC m=+231.389896773" watchObservedRunningTime="2026-07-30 15:13:24.155710164 +0000 UTC m=+231.390841011" Jul 30 15:13:24.222909 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.222879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-serving-certs-ca-bundle\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.223078 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.222915 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-telemeter-client-tls\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.223078 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.222933 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-federate-client-tls\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.223253 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.223206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-secret-telemeter-client\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.223494 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.223455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.223587 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.223552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-metrics-client-ca\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.223587 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.223573 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.223671 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.223604 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75hhq\" (UniqueName: \"kubernetes.io/projected/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-kube-api-access-75hhq\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.324360 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.324289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-metrics-client-ca\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.324360 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.324333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.324360 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.324357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75hhq\" (UniqueName: \"kubernetes.io/projected/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-kube-api-access-75hhq\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.324605 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.324488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-serving-certs-ca-bundle\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.324605 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.324519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-telemeter-client-tls\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.324704 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.324658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-federate-client-tls\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.324767 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.324740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-secret-telemeter-client\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.324850 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.324828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.325117 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.325089 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-metrics-client-ca\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.325262 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.325180 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-serving-certs-ca-bundle\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.325860 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.325831 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.327190 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.327173 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-federate-client-tls\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.327412 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.327388 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-secret-telemeter-client\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.327412 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.327406 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-telemeter-client-tls\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.327527 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.327420 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.333673 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.333654 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75hhq\" (UniqueName: \"kubernetes.io/projected/f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6-kube-api-access-75hhq\") pod \"telemeter-client-8466886869-jwkvt\" (UID: \"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6\") " pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.383355 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.383325 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" Jul 30 15:13:24.388386 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.388361 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 15:13:24.388714 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.388637 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="prometheus" containerID="cri-o://377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa" gracePeriod=600 Jul 30 15:13:24.388714 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.388650 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="kube-rbac-proxy" containerID="cri-o://172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57" gracePeriod=600 Jul 30 15:13:24.388714 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.388691 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="config-reloader" containerID="cri-o://4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb" gracePeriod=600 Jul 30 15:13:24.388947 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.388678 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0" gracePeriod=600 Jul 30 15:13:24.388947 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.388704 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="kube-rbac-proxy-web" containerID="cri-o://0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36" gracePeriod=600 Jul 30 15:13:24.388947 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.388679 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="thanos-sidecar" containerID="cri-o://521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12" gracePeriod=600 Jul 30 15:13:24.557768 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.557736 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8466886869-jwkvt"] Jul 30 15:13:24.560861 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:13:24.560834 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c6a7b9_75b5_46dd_a5ac_a7449b3affd6.slice/crio-e025066a2efe6665a48522054719895f3cd4fa7a03511421c3316104c1d6eb71 WatchSource:0}: Error finding container e025066a2efe6665a48522054719895f3cd4fa7a03511421c3316104c1d6eb71: Status 404 returned error can't find the container with id e025066a2efe6665a48522054719895f3cd4fa7a03511421c3316104c1d6eb71 Jul 30 15:13:24.642369 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.642107 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:24.829291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829260 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.829291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829296 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc0f4712-e8fc-40c0-9e02-96996498bf5e-tls-assets\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.829508 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829353 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-trusted-ca-bundle\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.829508 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829374 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-thanos-prometheus-http-client-file\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.829508 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829404 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-kube-rbac-proxy\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.829508 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829433 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcmp8\" (UniqueName: \"kubernetes.io/projected/fc0f4712-e8fc-40c0-9e02-96996498bf5e-kube-api-access-dcmp8\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.829508 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829466 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-prometheus-k8s-tls\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.829508 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829499 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-config\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.829821 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829526 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-rulefiles-1\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.829821 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829557 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-configmap-metrics-client-ca\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.829821 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829585 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-configmap-kubelet-serving-ca-bundle\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.829821 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829610 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-metrics-client-certs\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.829821 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829645 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-configmap-serving-certs-ca-bundle\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.829821 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829674 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-grpc-tls\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.829821 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829701 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc0f4712-e8fc-40c0-9e02-96996498bf5e-config-out\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.829821 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829735 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-rulefiles-2\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.829821 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829781 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.830264 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829865 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-rulefiles-0\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.830264 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829867 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:13:24.830264 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829892 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-db\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.830264 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.829916 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-web-config\") pod \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\" (UID: \"fc0f4712-e8fc-40c0-9e02-96996498bf5e\") " Jul 30 15:13:24.830264 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.830147 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:13:24.830264 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.830214 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.830264 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.830233 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.830931 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.830900 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:13:24.831130 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.831108 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:13:24.831272 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.831253 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:13:24.831348 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.831329 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:13:24.832230 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.832195 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:13:24.832336 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.832284 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:13:24.832336 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.832297 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0f4712-e8fc-40c0-9e02-96996498bf5e-kube-api-access-dcmp8" (OuterVolumeSpecName: "kube-api-access-dcmp8") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "kube-api-access-dcmp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:13:24.832623 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.832600 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:13:24.832907 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.832791 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:13:24.833270 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.833104 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0f4712-e8fc-40c0-9e02-96996498bf5e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:13:24.833270 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.833233 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:13:24.833856 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.833834 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:13:24.833966 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.833948 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0f4712-e8fc-40c0-9e02-96996498bf5e-config-out" (OuterVolumeSpecName: "config-out") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:13:24.834201 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.834170 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:13:24.834284 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.834211 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:13:24.834284 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.834226 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:13:24.835020 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.834998 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-config" (OuterVolumeSpecName: "config") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:13:24.844586 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.844565 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-web-config" (OuterVolumeSpecName: "web-config") pod "fc0f4712-e8fc-40c0-9e02-96996498bf5e" (UID: "fc0f4712-e8fc-40c0-9e02-96996498bf5e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:13:24.931032 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.930977 2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.931032 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.930998 2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-kube-rbac-proxy\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.931032 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.931008 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcmp8\" (UniqueName: \"kubernetes.io/projected/fc0f4712-e8fc-40c0-9e02-96996498bf5e-kube-api-access-dcmp8\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.931032 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.931016 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.931032 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.931027 2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-config\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.931230 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.931036 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.931230 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.931044 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-configmap-metrics-client-ca\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.931230 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.931053 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.931230 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.931061 2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-metrics-client-certs\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.931230 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.931070 2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-grpc-tls\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.931230 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.931078 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc0f4712-e8fc-40c0-9e02-96996498bf5e-config-out\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.931230 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.931087 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.931230 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.931095 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.931230 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.931104 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.931230 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.931112 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc0f4712-e8fc-40c0-9e02-96996498bf5e-prometheus-k8s-db\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.931230 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.931121 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-web-config\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.931230 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.931129 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc0f4712-e8fc-40c0-9e02-96996498bf5e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:24.931230 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:24.931137 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc0f4712-e8fc-40c0-9e02-96996498bf5e-tls-assets\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:13:25.103142 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.103113 2579 generic.go:356] "Generic (PLEG): container finished" podID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerID="179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0" exitCode=0 Jul 30 15:13:25.103142 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.103138 2579 generic.go:356] "Generic (PLEG): container finished" podID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerID="172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57" exitCode=0 Jul 30 15:13:25.103142 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.103145 2579 generic.go:356] "Generic (PLEG): container finished" podID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerID="0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36" exitCode=0 Jul 30 15:13:25.103601 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.103150 2579 generic.go:356] "Generic (PLEG): container finished" podID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerID="521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12" exitCode=0 Jul 30 15:13:25.103601 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.103157 2579 generic.go:356] "Generic (PLEG): container finished" podID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerID="4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb" exitCode=0 Jul 30 15:13:25.103601 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.103163 2579 generic.go:356] "Generic (PLEG): container finished" podID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerID="377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa" exitCode=0 Jul 30 15:13:25.103601 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.103191 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.103601 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.103198 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc0f4712-e8fc-40c0-9e02-96996498bf5e","Type":"ContainerDied","Data":"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0"} Jul 30 15:13:25.103601 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.103272 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc0f4712-e8fc-40c0-9e02-96996498bf5e","Type":"ContainerDied","Data":"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57"} Jul 30 15:13:25.103601 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.103313 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc0f4712-e8fc-40c0-9e02-96996498bf5e","Type":"ContainerDied","Data":"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36"} Jul 30 15:13:25.103601 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.103344 2579 scope.go:117] "RemoveContainer" containerID="179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0" Jul 30 15:13:25.103601 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.103348 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc0f4712-e8fc-40c0-9e02-96996498bf5e","Type":"ContainerDied","Data":"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12"} Jul 30 15:13:25.103601 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.103468 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc0f4712-e8fc-40c0-9e02-96996498bf5e","Type":"ContainerDied","Data":"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb"} Jul 30 15:13:25.103601 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.103504 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc0f4712-e8fc-40c0-9e02-96996498bf5e","Type":"ContainerDied","Data":"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa"} Jul 30 15:13:25.103601 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.103542 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc0f4712-e8fc-40c0-9e02-96996498bf5e","Type":"ContainerDied","Data":"7a03f80a485f670be569e628c39987ef1cf902df86bad9c7567e18bd872e5b3d"} Jul 30 15:13:25.104567 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.104545 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" event={"ID":"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6","Type":"ContainerStarted","Data":"e025066a2efe6665a48522054719895f3cd4fa7a03511421c3316104c1d6eb71"} Jul 30 15:13:25.111121 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.111030 2579 scope.go:117] "RemoveContainer" containerID="172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57" Jul 30 15:13:25.117764 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.117747 2579 scope.go:117] "RemoveContainer" containerID="0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36" Jul 30 15:13:25.124286 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.124261 2579 scope.go:117] "RemoveContainer" containerID="521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12" Jul 30 15:13:25.130597 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.130581 2579 scope.go:117] "RemoveContainer" containerID="4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb" Jul 30 15:13:25.136602 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.136557 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 15:13:25.138105 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.138085 2579 scope.go:117] "RemoveContainer" containerID="377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa" Jul 30 15:13:25.142404 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.142375 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 15:13:25.145296 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.145276 2579 scope.go:117] "RemoveContainer" containerID="95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45" Jul 30 15:13:25.151521 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.151502 2579 scope.go:117] "RemoveContainer" containerID="179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0" Jul 30 15:13:25.151767 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:13:25.151749 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0\": container with ID starting with 179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0 not found: ID does not exist" containerID="179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0" Jul 30 15:13:25.151842 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.151775 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0"} err="failed to get container status \"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0\": rpc error: code = NotFound desc = could not find container \"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0\": container with ID starting with 179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0 not found: ID does not exist" Jul 30 15:13:25.151842 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.151792 2579 scope.go:117] "RemoveContainer" containerID="172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57" Jul 30 15:13:25.152045 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:13:25.152026 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57\": container with ID starting with 172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57 not found: ID does not exist" containerID="172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57" Jul 30 15:13:25.152104 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.152053 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57"} err="failed to get container status \"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57\": rpc error: code = NotFound desc = could not find container \"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57\": container with ID starting with 172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57 not found: ID does not exist" Jul 30 15:13:25.152104 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.152075 2579 scope.go:117] "RemoveContainer" containerID="0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36" Jul 30 15:13:25.152294 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:13:25.152277 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36\": container with ID starting with 0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36 not found: ID does not exist" containerID="0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36" Jul 30 15:13:25.152331 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.152298 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36"} err="failed to get container status \"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36\": rpc error: code = NotFound desc = could not find container \"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36\": container with ID starting with 0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36 not found: ID does not exist" Jul 30 15:13:25.152331 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.152312 2579 scope.go:117] "RemoveContainer" containerID="521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12" Jul 30 15:13:25.152534 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:13:25.152517 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12\": container with ID starting with 521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12 not found: ID does not exist" containerID="521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12" Jul 30 15:13:25.152592 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.152540 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12"} err="failed to get container status \"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12\": rpc error: code = NotFound desc = could not find container \"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12\": container with ID starting with 521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12 not found: ID does not exist" Jul 30 15:13:25.152592 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.152561 2579 scope.go:117] "RemoveContainer" containerID="4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb" Jul 30 15:13:25.152781 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:13:25.152765 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb\": container with ID starting with 4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb not found: ID does not exist" containerID="4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb" Jul 30 15:13:25.152921 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.152785 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb"} err="failed to get container status \"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb\": rpc error: code = NotFound desc = could not find container \"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb\": container with ID starting with 4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb not found: ID does not exist" Jul 30 15:13:25.152921 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.152811 2579 scope.go:117] "RemoveContainer" containerID="377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa" Jul 30 15:13:25.153039 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:13:25.153022 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa\": container with ID starting with 377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa not found: ID does not exist" containerID="377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa" Jul 30 15:13:25.153093 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.153044 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa"} err="failed to get container status \"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa\": rpc error: code = NotFound desc = could not find container \"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa\": container with ID starting with 377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa not found: ID does not exist" Jul 30 15:13:25.153093 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.153058 2579 scope.go:117] "RemoveContainer" containerID="95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45" Jul 30 15:13:25.153265 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:13:25.153248 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45\": container with ID starting with 95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45 not found: ID does not exist" containerID="95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45" Jul 30 15:13:25.153299 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.153269 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45"} err="failed to get container status \"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45\": rpc error: code = NotFound desc = could not find container \"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45\": container with ID starting with 95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45 not found: ID does not exist" Jul 30 15:13:25.153299 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.153284 2579 scope.go:117] "RemoveContainer" containerID="179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0" Jul 30 15:13:25.153503 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.153479 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0"} err="failed to get container status \"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0\": rpc error: code = NotFound desc = could not find container \"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0\": container with ID starting with 179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0 not found: ID does not exist" Jul 30 15:13:25.153544 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.153504 2579 scope.go:117] "RemoveContainer" containerID="172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57" Jul 30 15:13:25.153692 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.153673 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57"} err="failed to get container status \"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57\": rpc error: code = NotFound desc = could not find container \"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57\": container with ID starting with 172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57 not found: ID does not exist" Jul 30 15:13:25.153692 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.153690 2579 scope.go:117] "RemoveContainer" containerID="0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36" Jul 30 15:13:25.153882 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.153859 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36"} err="failed to get container status \"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36\": rpc error: code = NotFound desc = could not find container \"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36\": container with ID starting with 0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36 not found: ID does not exist" Jul 30 15:13:25.153923 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.153884 2579 scope.go:117] "RemoveContainer" containerID="521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12" Jul 30 15:13:25.154039 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.154023 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12"} err="failed to get container status \"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12\": rpc error: code = NotFound desc = could not find container \"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12\": container with ID starting with 521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12 not found: ID does not exist" Jul 30 15:13:25.154075 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.154039 2579 scope.go:117] "RemoveContainer" containerID="4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb" Jul 30 15:13:25.154228 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.154211 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb"} err="failed to get container status \"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb\": rpc error: code = NotFound desc = could not find container \"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb\": container with ID starting with 4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb not found: ID does not exist" Jul 30 15:13:25.154274 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.154229 2579 scope.go:117] "RemoveContainer" containerID="377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa" Jul 30 15:13:25.154435 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.154417 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa"} err="failed to get container status \"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa\": rpc error: code = NotFound desc = could not find container \"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa\": container with ID starting with 377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa not found: ID does not exist" Jul 30 15:13:25.154507 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.154437 2579 scope.go:117] "RemoveContainer" containerID="95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45" Jul 30 15:13:25.154626 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.154610 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45"} err="failed to get container status \"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45\": rpc error: code = NotFound desc = could not find container \"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45\": container with ID starting with 95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45 not found: ID does not exist" Jul 30 15:13:25.154670 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.154627 2579 scope.go:117] "RemoveContainer" containerID="179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0" Jul 30 15:13:25.154823 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.154790 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0"} err="failed to get container status \"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0\": rpc error: code = NotFound desc = could not find container \"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0\": container with ID starting with 179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0 not found: ID does not exist" Jul 30 15:13:25.154888 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.154825 2579 scope.go:117] "RemoveContainer" containerID="172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57" Jul 30 15:13:25.155057 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.155041 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57"} err="failed to get container status \"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57\": rpc error: code = NotFound desc = could not find container \"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57\": container with ID starting with 172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57 not found: ID does not exist" Jul 30 15:13:25.155097 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.155058 2579 scope.go:117] "RemoveContainer" containerID="0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36" Jul 30 15:13:25.155237 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.155222 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36"} err="failed to get container status \"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36\": rpc error: code = NotFound desc = could not find container \"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36\": container with ID starting with 0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36 not found: ID does not exist" Jul 30 15:13:25.155237 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.155236 2579 scope.go:117] "RemoveContainer" containerID="521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12" Jul 30 15:13:25.155443 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.155425 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12"} err="failed to get container status \"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12\": rpc error: code = NotFound desc = could not find container \"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12\": container with ID starting with 521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12 not found: ID does not exist" Jul 30 15:13:25.155480 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.155446 2579 scope.go:117] "RemoveContainer" containerID="4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb" Jul 30 15:13:25.155652 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.155631 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb"} err="failed to get container status \"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb\": rpc error: code = NotFound desc = could not find container \"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb\": container with ID starting with 4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb not found: ID does not exist" Jul 30 15:13:25.155652 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.155649 2579 scope.go:117] "RemoveContainer" containerID="377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa" Jul 30 15:13:25.155874 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.155854 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa"} err="failed to get container status \"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa\": rpc error: code = NotFound desc = could not find container \"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa\": container with ID starting with 377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa not found: ID does not exist" Jul 30 15:13:25.155940 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.155875 2579 scope.go:117] "RemoveContainer" containerID="95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45" Jul 30 15:13:25.156064 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.156046 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45"} err="failed to get container status \"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45\": rpc error: code = NotFound desc = could not find container \"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45\": container with ID starting with 95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45 not found: ID does not exist" Jul 30 15:13:25.156102 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.156064 2579 scope.go:117] "RemoveContainer" containerID="179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0" Jul 30 15:13:25.156246 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.156227 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0"} err="failed to get container status \"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0\": rpc error: code = NotFound desc = could not find container \"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0\": container with ID starting with 179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0 not found: ID does not exist" Jul 30 15:13:25.156286 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.156248 2579 scope.go:117] "RemoveContainer" containerID="172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57" Jul 30 15:13:25.156464 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.156444 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57"} err="failed to get container status \"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57\": rpc error: code = NotFound desc = could not find container \"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57\": container with ID starting with 172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57 not found: ID does not exist" Jul 30 15:13:25.156535 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.156466 2579 scope.go:117] "RemoveContainer" containerID="0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36" Jul 30 15:13:25.156686 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.156669 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36"} err="failed to get container status \"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36\": rpc error: code = NotFound desc = could not find container \"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36\": container with ID starting with 0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36 not found: ID does not exist" Jul 30 15:13:25.156730 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.156687 2579 scope.go:117] "RemoveContainer" containerID="521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12" Jul 30 15:13:25.156886 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.156869 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12"} err="failed to get container status \"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12\": rpc error: code = NotFound desc = could not find container \"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12\": container with ID starting with 521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12 not found: ID does not exist" Jul 30 15:13:25.156955 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.156888 2579 scope.go:117] "RemoveContainer" containerID="4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb" Jul 30 15:13:25.157121 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.157100 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb"} err="failed to get container status \"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb\": rpc error: code = NotFound desc = could not find container \"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb\": container with ID starting with 4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb not found: ID does not exist" Jul 30 15:13:25.157170 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.157122 2579 scope.go:117] "RemoveContainer" containerID="377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa" Jul 30 15:13:25.157327 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.157310 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa"} err="failed to get container status \"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa\": rpc error: code = NotFound desc = could not find container \"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa\": container with ID starting with 377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa not found: ID does not exist" Jul 30 15:13:25.157392 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.157328 2579 scope.go:117] "RemoveContainer" containerID="95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45" Jul 30 15:13:25.157535 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.157518 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45"} err="failed to get container status \"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45\": rpc error: code = NotFound desc = could not find container \"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45\": container with ID starting with 95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45 not found: ID does not exist" Jul 30 15:13:25.157576 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.157535 2579 scope.go:117] "RemoveContainer" containerID="179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0" Jul 30 15:13:25.157773 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.157755 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0"} err="failed to get container status \"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0\": rpc error: code = NotFound desc = could not find container \"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0\": container with ID starting with 179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0 not found: ID does not exist" Jul 30 15:13:25.157848 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.157774 2579 scope.go:117] "RemoveContainer" containerID="172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57" Jul 30 15:13:25.158030 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.158012 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57"} err="failed to get container status \"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57\": rpc error: code = NotFound desc = could not find container \"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57\": container with ID starting with 172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57 not found: ID does not exist" Jul 30 15:13:25.158076 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.158031 2579 scope.go:117] "RemoveContainer" containerID="0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36" Jul 30 15:13:25.158229 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.158212 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36"} err="failed to get container status \"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36\": rpc error: code = NotFound desc = could not find container \"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36\": container with ID starting with 0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36 not found: ID does not exist" Jul 30 15:13:25.158289 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.158230 2579 scope.go:117] "RemoveContainer" containerID="521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12" Jul 30 15:13:25.158424 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.158409 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12"} err="failed to get container status \"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12\": rpc error: code = NotFound desc = could not find container \"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12\": container with ID starting with 521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12 not found: ID does not exist" Jul 30 15:13:25.158467 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.158426 2579 scope.go:117] "RemoveContainer" containerID="4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb" Jul 30 15:13:25.158617 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.158600 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb"} err="failed to get container status \"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb\": rpc error: code = NotFound desc = could not find container \"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb\": container with ID starting with 4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb not found: ID does not exist" Jul 30 15:13:25.158676 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.158618 2579 scope.go:117] "RemoveContainer" containerID="377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa" Jul 30 15:13:25.158840 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.158821 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa"} err="failed to get container status \"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa\": rpc error: code = NotFound desc = could not find container \"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa\": container with ID starting with 377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa not found: ID does not exist" Jul 30 15:13:25.158840 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.158838 2579 scope.go:117] "RemoveContainer" containerID="95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45" Jul 30 15:13:25.159032 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.159016 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45"} err="failed to get container status \"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45\": rpc error: code = NotFound desc = could not find container \"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45\": container with ID starting with 95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45 not found: ID does not exist" Jul 30 15:13:25.159078 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.159033 2579 scope.go:117] "RemoveContainer" containerID="179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0" Jul 30 15:13:25.159234 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.159218 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0"} err="failed to get container status \"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0\": rpc error: code = NotFound desc = could not find container \"179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0\": container with ID starting with 179e584c9c01c0b71cb4e898dd74b2f61c84a5ee2c6cd7baabd82367c29519f0 not found: ID does not exist" Jul 30 15:13:25.159313 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.159233 2579 scope.go:117] "RemoveContainer" containerID="172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57" Jul 30 15:13:25.159446 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.159428 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57"} err="failed to get container status \"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57\": rpc error: code = NotFound desc = could not find container \"172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57\": container with ID starting with 172aeb1f4e2880cde56f9be774368580a5b7d34adfc7a1345ef3c22faac76c57 not found: ID does not exist" Jul 30 15:13:25.159487 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.159447 2579 scope.go:117] "RemoveContainer" containerID="0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36" Jul 30 15:13:25.159644 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.159629 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36"} err="failed to get container status \"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36\": rpc error: code = NotFound desc = could not find container \"0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36\": container with ID starting with 0749d93fb075505ed0b49e3734c6a7f2a9af5f18cac80b93929657f94f79cc36 not found: ID does not exist" Jul 30 15:13:25.159687 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.159644 2579 scope.go:117] "RemoveContainer" containerID="521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12" Jul 30 15:13:25.159836 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.159820 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12"} err="failed to get container status \"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12\": rpc error: code = NotFound desc = could not find container \"521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12\": container with ID starting with 521b5dd7175b173ea0e755741f603526df1b91803d3489cec2e27cbfb1b24c12 not found: ID does not exist" Jul 30 15:13:25.159874 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.159836 2579 scope.go:117] "RemoveContainer" containerID="4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb" Jul 30 15:13:25.160042 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.160013 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb"} err="failed to get container status \"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb\": rpc error: code = NotFound desc = could not find container \"4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb\": container with ID starting with 4ac505551b837fc7ec1ee3d1f91ab61c7c0ac725390c83354f452fa4e347f5cb not found: ID does not exist" Jul 30 15:13:25.160042 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.160040 2579 scope.go:117] "RemoveContainer" containerID="377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa" Jul 30 15:13:25.160234 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.160218 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa"} err="failed to get container status \"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa\": rpc error: code = NotFound desc = could not find container \"377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa\": container with ID starting with 377b9df016430531bcb0caa072bc5e1283a5bfbcf3d52f0e04c5028cd04452fa not found: ID does not exist" Jul 30 15:13:25.160275 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.160235 2579 scope.go:117] "RemoveContainer" containerID="95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45" Jul 30 15:13:25.160429 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.160413 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45"} err="failed to get container status \"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45\": rpc error: code = NotFound desc = could not find container \"95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45\": container with ID starting with 95cc929e0377e9f5cf56c5dce325ecc2fffa1db20e1e74eb22b6a76de8cfff45 not found: ID does not exist" Jul 30 15:13:25.178424 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178403 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 15:13:25.178708 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178695 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="init-config-reloader" Jul 30 15:13:25.178748 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178710 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="init-config-reloader" Jul 30 15:13:25.178748 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178718 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="thanos-sidecar" Jul 30 15:13:25.178748 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178724 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="thanos-sidecar" Jul 30 15:13:25.178748 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178735 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="kube-rbac-proxy-thanos" Jul 30 15:13:25.178748 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178741 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="kube-rbac-proxy-thanos" Jul 30 15:13:25.178936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178751 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="kube-rbac-proxy" Jul 30 15:13:25.178936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178757 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="kube-rbac-proxy" Jul 30 15:13:25.178936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178764 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="config-reloader" Jul 30 15:13:25.178936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178769 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="config-reloader" Jul 30 15:13:25.178936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178776 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="kube-rbac-proxy-web" Jul 30 15:13:25.178936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178781 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="kube-rbac-proxy-web" Jul 30 15:13:25.178936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178789 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="prometheus" Jul 30 15:13:25.178936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178809 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="prometheus" Jul 30 15:13:25.178936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178859 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="thanos-sidecar" Jul 30 15:13:25.178936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178867 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="prometheus" Jul 30 15:13:25.178936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178872 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="kube-rbac-proxy-web" Jul 30 15:13:25.178936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178879 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="config-reloader" Jul 30 15:13:25.178936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178886 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="kube-rbac-proxy-thanos" Jul 30 15:13:25.178936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.178892 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" containerName="kube-rbac-proxy" Jul 30 15:13:25.182969 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.182927 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.186847 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.186829 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 30 15:13:25.186936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.186857 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 30 15:13:25.186936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.186907 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 30 15:13:25.187214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.187199 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 30 15:13:25.188101 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.188086 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 30 15:13:25.189600 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.189461 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 30 15:13:25.189830 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.189812 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 30 15:13:25.190066 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.190047 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5zfp2\"" Jul 30 15:13:25.190345 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.190331 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 30 15:13:25.190628 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.190611 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8dn27m6fkh44t\"" Jul 30 15:13:25.191499 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.191469 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 30 15:13:25.192090 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.191978 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 30 15:13:25.192090 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.191982 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 30 15:13:25.192090 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.192017 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 30 15:13:25.194246 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.194225 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 30 15:13:25.201093 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.201064 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 30 15:13:25.202434 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.202417 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc0f4712-e8fc-40c0-9e02-96996498bf5e" path="/var/lib/kubelet/pods/fc0f4712-e8fc-40c0-9e02-96996498bf5e/volumes" Jul 30 15:13:25.208563 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.208472 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 15:13:25.334820 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.334779 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dded69a3-a4db-4f1c-8102-ccc1539a4118-config-out\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.334955 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.334823 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.334955 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.334853 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.334955 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.334878 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk27f\" (UniqueName: \"kubernetes.io/projected/dded69a3-a4db-4f1c-8102-ccc1539a4118-kube-api-access-lk27f\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.334955 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.334924 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.335154 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.334995 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dded69a3-a4db-4f1c-8102-ccc1539a4118-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.335154 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.335035 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.335154 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.335057 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dded69a3-a4db-4f1c-8102-ccc1539a4118-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.335154 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.335087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.335358 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.335158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.335358 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.335203 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.335358 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.335245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.335358 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.335291 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-web-config\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.335358 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.335319 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-config\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.335536 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.335366 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.335536 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.335392 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.335536 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.335422 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.335536 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.335451 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.335536 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.335491 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.335705 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.335545 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.436221 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436147 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.436221 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.436221 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436203 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.436550 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.436550 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.436550 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.436550 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dded69a3-a4db-4f1c-8102-ccc1539a4118-config-out\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.436550 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.436550 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.436550 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk27f\" (UniqueName: \"kubernetes.io/projected/dded69a3-a4db-4f1c-8102-ccc1539a4118-kube-api-access-lk27f\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.436550 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436415 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.436550 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436469 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dded69a3-a4db-4f1c-8102-ccc1539a4118-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.436550 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.436550 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dded69a3-a4db-4f1c-8102-ccc1539a4118-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.437140 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436579 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.437140 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.437140 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.437140 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.437140 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-web-config\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.437140 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-config\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.437140 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.436947 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.439387 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.439361 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.439387 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.439360 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.439543 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.439512 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dded69a3-a4db-4f1c-8102-ccc1539a4118-config-out\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.439621 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.439590 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.439761 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.439738 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.440692 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.440159 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-web-config\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.440692 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.440252 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.440692 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.440310 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.440692 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.440398 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.440692 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.440417 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dded69a3-a4db-4f1c-8102-ccc1539a4118-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.440692 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.440685 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.441042 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.440872 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.441096 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.441066 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.441508 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.441482 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dded69a3-a4db-4f1c-8102-ccc1539a4118-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.442086 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.442065 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dded69a3-a4db-4f1c-8102-ccc1539a4118-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.442306 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.442286 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-config\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.442744 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.442724 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.442999 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.442981 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dded69a3-a4db-4f1c-8102-ccc1539a4118-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.452101 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.452080 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk27f\" (UniqueName: \"kubernetes.io/projected/dded69a3-a4db-4f1c-8102-ccc1539a4118-kube-api-access-lk27f\") pod \"prometheus-k8s-0\" (UID: \"dded69a3-a4db-4f1c-8102-ccc1539a4118\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.492978 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.492958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:13:25.655441 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:25.655414 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 15:13:25.656435 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:13:25.656410 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddded69a3_a4db_4f1c_8102_ccc1539a4118.slice/crio-c5778b46cfb0e807c2317e34c0ca9e4b681e480134ff05a8b4d399d624ef0758 WatchSource:0}: Error finding container c5778b46cfb0e807c2317e34c0ca9e4b681e480134ff05a8b4d399d624ef0758: Status 404 returned error can't find the container with id c5778b46cfb0e807c2317e34c0ca9e4b681e480134ff05a8b4d399d624ef0758 Jul 30 15:13:26.110611 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:26.110533 2579 generic.go:356] "Generic (PLEG): container finished" podID="dded69a3-a4db-4f1c-8102-ccc1539a4118" containerID="c5e4cf527f02bfc5a9f53adc3a579e94aa4ba9364477260e4c21c1a3ab9ac50d" exitCode=0 Jul 30 15:13:26.111025 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:26.110635 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dded69a3-a4db-4f1c-8102-ccc1539a4118","Type":"ContainerDied","Data":"c5e4cf527f02bfc5a9f53adc3a579e94aa4ba9364477260e4c21c1a3ab9ac50d"} Jul 30 15:13:26.111025 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:26.110720 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dded69a3-a4db-4f1c-8102-ccc1539a4118","Type":"ContainerStarted","Data":"c5778b46cfb0e807c2317e34c0ca9e4b681e480134ff05a8b4d399d624ef0758"} Jul 30 15:13:27.119260 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:27.119231 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dded69a3-a4db-4f1c-8102-ccc1539a4118","Type":"ContainerStarted","Data":"5a3d68a2ebbfe468f9e57f495eccae66ab3c6ae47afb6493f799b57d78bb39f9"} Jul 30 15:13:27.119652 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:27.119307 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dded69a3-a4db-4f1c-8102-ccc1539a4118","Type":"ContainerStarted","Data":"148768b2ceb4cf709eb159157be38f467eed0988c481aa29a0124659bb81913d"} Jul 30 15:13:27.119652 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:27.119361 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dded69a3-a4db-4f1c-8102-ccc1539a4118","Type":"ContainerStarted","Data":"6faef9fccb80ef487634d531fec1cb69d9ef1aeda0ecb1f1c77ac8f4ecc555fc"} Jul 30 15:13:27.119652 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:27.119402 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dded69a3-a4db-4f1c-8102-ccc1539a4118","Type":"ContainerStarted","Data":"077975368c10514ee0f156dc39eaba8ff372bc405e3a94009e0cff3195f4ae68"} Jul 30 15:13:27.119652 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:27.119441 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dded69a3-a4db-4f1c-8102-ccc1539a4118","Type":"ContainerStarted","Data":"e5c8f1e0afa3dfcdc329df3fb98a9710e9423b3941c2c3ea07062da425c1e598"} Jul 30 15:13:27.119652 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:27.119484 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dded69a3-a4db-4f1c-8102-ccc1539a4118","Type":"ContainerStarted","Data":"4588f2c83e2f3fae043839270f63b7717d07d3f289f71d563a891e2bf1aedf46"} Jul 30 15:13:27.126339 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:27.122114 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" event={"ID":"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6","Type":"ContainerStarted","Data":"e0de847a4c66623804a9b4297d14fa889fcc01059c026e22b212d465bd54811c"} Jul 30 15:13:27.126339 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:27.122227 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" event={"ID":"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6","Type":"ContainerStarted","Data":"6dbe508524d630201f6ea798fa2bdb2b29434ae773cca8f4a8314457ed5de067"} Jul 30 15:13:27.126339 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:27.122267 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" event={"ID":"f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6","Type":"ContainerStarted","Data":"7e331b5238cf90234fd24da27ed411ad0f77218602c80ed0dfa227aba2364c7c"} Jul 30 15:13:27.162402 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:27.162358 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.143064813 podStartE2EDuration="2.162345464s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.280651ms" isStatefulPod=false podCreationTimestamp="2026-07-30 15:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:13:27.160997121 +0000 UTC m=+234.396127968" watchObservedRunningTime="2026-07-30 15:13:27.162345464 +0000 UTC m=+234.397476309" Jul 30 15:13:27.194405 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:27.194363 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-8466886869-jwkvt" podStartSLOduration=0.880368912 podStartE2EDuration="3.194345252s" totalImagesPullingTime="2.31397634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:13:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:13:27.191206013 +0000 UTC m=+234.426336859" watchObservedRunningTime="2026-07-30 15:13:27.194345252 +0000 UTC m=+234.429476099" Jul 30 15:13:30.493349 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:13:30.493308 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:14:25.494150 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:14:25.494114 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:14:25.509162 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:14:25.509141 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:14:26.361763 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:14:26.361736 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 15:14:33.144671 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:14:33.144644 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 30 15:16:20.499358 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:20.499330 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-c9dgq"] Jul 30 15:16:20.502514 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:20.502490 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-c9dgq" Jul 30 15:16:20.505596 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:20.505572 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 30 15:16:20.506951 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:20.506935 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-4kcrc\"" Jul 30 15:16:20.507056 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:20.506939 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 30 15:16:20.514163 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:20.514143 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-c9dgq"] Jul 30 15:16:20.633525 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:20.633501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjrn7\" (UniqueName: \"kubernetes.io/projected/ea200af0-1bd4-48bb-af7a-85ab38c767e0-kube-api-access-rjrn7\") pod \"cert-manager-55b4d954d5-c9dgq\" (UID: \"ea200af0-1bd4-48bb-af7a-85ab38c767e0\") " pod="cert-manager/cert-manager-55b4d954d5-c9dgq" Jul 30 15:16:20.633619 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:20.633536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea200af0-1bd4-48bb-af7a-85ab38c767e0-bound-sa-token\") pod \"cert-manager-55b4d954d5-c9dgq\" (UID: \"ea200af0-1bd4-48bb-af7a-85ab38c767e0\") " pod="cert-manager/cert-manager-55b4d954d5-c9dgq" Jul 30 15:16:20.734825 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:20.734787 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjrn7\" (UniqueName: \"kubernetes.io/projected/ea200af0-1bd4-48bb-af7a-85ab38c767e0-kube-api-access-rjrn7\") pod \"cert-manager-55b4d954d5-c9dgq\" (UID: \"ea200af0-1bd4-48bb-af7a-85ab38c767e0\") " pod="cert-manager/cert-manager-55b4d954d5-c9dgq" Jul 30 15:16:20.734899 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:20.734843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea200af0-1bd4-48bb-af7a-85ab38c767e0-bound-sa-token\") pod \"cert-manager-55b4d954d5-c9dgq\" (UID: \"ea200af0-1bd4-48bb-af7a-85ab38c767e0\") " pod="cert-manager/cert-manager-55b4d954d5-c9dgq" Jul 30 15:16:20.743611 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:20.743592 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea200af0-1bd4-48bb-af7a-85ab38c767e0-bound-sa-token\") pod \"cert-manager-55b4d954d5-c9dgq\" (UID: \"ea200af0-1bd4-48bb-af7a-85ab38c767e0\") " pod="cert-manager/cert-manager-55b4d954d5-c9dgq" Jul 30 15:16:20.743704 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:20.743665 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjrn7\" (UniqueName: \"kubernetes.io/projected/ea200af0-1bd4-48bb-af7a-85ab38c767e0-kube-api-access-rjrn7\") pod \"cert-manager-55b4d954d5-c9dgq\" (UID: \"ea200af0-1bd4-48bb-af7a-85ab38c767e0\") " pod="cert-manager/cert-manager-55b4d954d5-c9dgq" Jul 30 15:16:20.818395 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:20.818342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-c9dgq" Jul 30 15:16:20.941470 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:20.941447 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-c9dgq"] Jul 30 15:16:20.944534 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:16:20.944508 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea200af0_1bd4_48bb_af7a_85ab38c767e0.slice/crio-982e6e1e0c4b4030e670cf303a59779e1c94ee3c555984a6cde430345f215ad9 WatchSource:0}: Error finding container 982e6e1e0c4b4030e670cf303a59779e1c94ee3c555984a6cde430345f215ad9: Status 404 returned error can't find the container with id 982e6e1e0c4b4030e670cf303a59779e1c94ee3c555984a6cde430345f215ad9 Jul 30 15:16:20.946393 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:20.946376 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 15:16:21.773676 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:21.773623 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-c9dgq" event={"ID":"ea200af0-1bd4-48bb-af7a-85ab38c767e0","Type":"ContainerStarted","Data":"982e6e1e0c4b4030e670cf303a59779e1c94ee3c555984a6cde430345f215ad9"} Jul 30 15:16:24.788478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:24.788443 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-c9dgq" event={"ID":"ea200af0-1bd4-48bb-af7a-85ab38c767e0","Type":"ContainerStarted","Data":"9596b668afdf3a45b41b6ace2b5d15aa88952e7078519c3ab4816a3deeed421e"} Jul 30 15:16:24.807554 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:24.807509 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-c9dgq" podStartSLOduration=1.900844121 podStartE2EDuration="4.807493532s" totalImagesPullingTime="2.906649411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:16:24.805995231 +0000 UTC m=+412.041126075" watchObservedRunningTime="2026-07-30 15:16:24.807493532 +0000 UTC m=+412.042624371" Jul 30 15:16:47.330873 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.330841 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5"] Jul 30 15:16:47.336555 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.336533 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:16:47.341027 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.341008 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 30 15:16:47.341277 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.341260 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 30 15:16:47.341368 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.341351 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 30 15:16:47.341557 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.341544 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 30 15:16:47.341619 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.341546 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-t7drl\"" Jul 30 15:16:47.363029 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.363002 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5"] Jul 30 15:16:47.432419 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.432391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fa50e13-adca-4d63-9932-44713c9e0577-tmp\") pod \"opendatahub-operator-controller-manager-56444d5c7b-jxwk5\" (UID: \"7fa50e13-adca-4d63-9932-44713c9e0577\") " pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:16:47.432517 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.432425 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z694h\" (UniqueName: \"kubernetes.io/projected/7fa50e13-adca-4d63-9932-44713c9e0577-kube-api-access-z694h\") pod \"opendatahub-operator-controller-manager-56444d5c7b-jxwk5\" (UID: \"7fa50e13-adca-4d63-9932-44713c9e0577\") " pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:16:47.432595 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.432571 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fa50e13-adca-4d63-9932-44713c9e0577-webhook-cert\") pod \"opendatahub-operator-controller-manager-56444d5c7b-jxwk5\" (UID: \"7fa50e13-adca-4d63-9932-44713c9e0577\") " pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:16:47.432641 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.432615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7fa50e13-adca-4d63-9932-44713c9e0577-manifests\") pod \"opendatahub-operator-controller-manager-56444d5c7b-jxwk5\" (UID: \"7fa50e13-adca-4d63-9932-44713c9e0577\") " pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:16:47.432678 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.432641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7fa50e13-adca-4d63-9932-44713c9e0577-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56444d5c7b-jxwk5\" (UID: \"7fa50e13-adca-4d63-9932-44713c9e0577\") " pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:16:47.533687 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.533652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fa50e13-adca-4d63-9932-44713c9e0577-webhook-cert\") pod \"opendatahub-operator-controller-manager-56444d5c7b-jxwk5\" (UID: \"7fa50e13-adca-4d63-9932-44713c9e0577\") " pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:16:47.533687 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.533689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7fa50e13-adca-4d63-9932-44713c9e0577-manifests\") pod \"opendatahub-operator-controller-manager-56444d5c7b-jxwk5\" (UID: \"7fa50e13-adca-4d63-9932-44713c9e0577\") " pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:16:47.533936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.533707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7fa50e13-adca-4d63-9932-44713c9e0577-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56444d5c7b-jxwk5\" (UID: \"7fa50e13-adca-4d63-9932-44713c9e0577\") " pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:16:47.533936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.533769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fa50e13-adca-4d63-9932-44713c9e0577-tmp\") pod \"opendatahub-operator-controller-manager-56444d5c7b-jxwk5\" (UID: \"7fa50e13-adca-4d63-9932-44713c9e0577\") " pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:16:47.533936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.533791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z694h\" (UniqueName: \"kubernetes.io/projected/7fa50e13-adca-4d63-9932-44713c9e0577-kube-api-access-z694h\") pod \"opendatahub-operator-controller-manager-56444d5c7b-jxwk5\" (UID: \"7fa50e13-adca-4d63-9932-44713c9e0577\") " pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:16:47.534231 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.534191 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7fa50e13-adca-4d63-9932-44713c9e0577-manifests\") pod \"opendatahub-operator-controller-manager-56444d5c7b-jxwk5\" (UID: \"7fa50e13-adca-4d63-9932-44713c9e0577\") " pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:16:47.536153 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.536133 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7fa50e13-adca-4d63-9932-44713c9e0577-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56444d5c7b-jxwk5\" (UID: \"7fa50e13-adca-4d63-9932-44713c9e0577\") " pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:16:47.536388 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.536369 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fa50e13-adca-4d63-9932-44713c9e0577-tmp\") pod \"opendatahub-operator-controller-manager-56444d5c7b-jxwk5\" (UID: \"7fa50e13-adca-4d63-9932-44713c9e0577\") " pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:16:47.536587 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.536572 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fa50e13-adca-4d63-9932-44713c9e0577-webhook-cert\") pod \"opendatahub-operator-controller-manager-56444d5c7b-jxwk5\" (UID: \"7fa50e13-adca-4d63-9932-44713c9e0577\") " pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:16:47.544244 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.544222 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z694h\" (UniqueName: \"kubernetes.io/projected/7fa50e13-adca-4d63-9932-44713c9e0577-kube-api-access-z694h\") pod \"opendatahub-operator-controller-manager-56444d5c7b-jxwk5\" (UID: \"7fa50e13-adca-4d63-9932-44713c9e0577\") " pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:16:47.647627 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.647591 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:16:47.788669 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.788572 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5"] Jul 30 15:16:47.790755 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:16:47.790728 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa50e13_adca_4d63_9932_44713c9e0577.slice/crio-435b83d5975bf8697dcd6d9c434572be64c9b755733871873d6a0195a5eed53c WatchSource:0}: Error finding container 435b83d5975bf8697dcd6d9c434572be64c9b755733871873d6a0195a5eed53c: Status 404 returned error can't find the container with id 435b83d5975bf8697dcd6d9c434572be64c9b755733871873d6a0195a5eed53c Jul 30 15:16:47.884084 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:47.884049 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" event={"ID":"7fa50e13-adca-4d63-9932-44713c9e0577","Type":"ContainerStarted","Data":"435b83d5975bf8697dcd6d9c434572be64c9b755733871873d6a0195a5eed53c"} Jul 30 15:16:50.899710 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:50.899675 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" event={"ID":"7fa50e13-adca-4d63-9932-44713c9e0577","Type":"ContainerStarted","Data":"f82e1a610f8ab37759ff4ff1645045cb733197dbd2585dee5961c7d8a462be56"} Jul 30 15:16:51.904630 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:51.904597 2579 generic.go:356] "Generic (PLEG): container finished" podID="7fa50e13-adca-4d63-9932-44713c9e0577" containerID="f82e1a610f8ab37759ff4ff1645045cb733197dbd2585dee5961c7d8a462be56" exitCode=0 Jul 30 15:16:51.904996 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:51.904674 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" event={"ID":"7fa50e13-adca-4d63-9932-44713c9e0577","Type":"ContainerDied","Data":"f82e1a610f8ab37759ff4ff1645045cb733197dbd2585dee5961c7d8a462be56"} Jul 30 15:16:52.910910 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:52.910873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" event={"ID":"7fa50e13-adca-4d63-9932-44713c9e0577","Type":"ContainerStarted","Data":"2691914cbe27aceaca7e1e27559649e01568ac0b9fd50cd20c722815c0ed0f9d"} Jul 30 15:16:52.911306 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:52.911019 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:16:52.938173 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:16:52.938122 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" podStartSLOduration=1.561167371 podStartE2EDuration="5.938106735s" totalImagesPullingTime="3.380246945s" totalInitContainerRuntime="996.692419ms" isStatefulPod=false podCreationTimestamp="2026-07-30 15:16:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:16:52.935929324 +0000 UTC m=+440.171060169" watchObservedRunningTime="2026-07-30 15:16:52.938106735 +0000 UTC m=+440.173237580" Jul 30 15:17:03.918307 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:03.918233 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-56444d5c7b-jxwk5" Jul 30 15:17:17.440877 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.440838 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-556f467f75-szpt6"] Jul 30 15:17:17.448424 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.448395 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" Jul 30 15:17:17.454846 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.454529 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 30 15:17:17.455991 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.455970 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 30 15:17:17.456090 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.456039 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 30 15:17:17.456090 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.456039 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 30 15:17:17.456190 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.456096 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 30 15:17:17.456190 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.456048 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vwb5q\"" Jul 30 15:17:17.473484 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.473458 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-556f467f75-szpt6"] Jul 30 15:17:17.508512 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.508481 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf"] Jul 30 15:17:17.511704 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.511686 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.520418 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.520392 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-zng5b\"" Jul 30 15:17:17.520418 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.520394 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 30 15:17:17.552439 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.552418 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf"] Jul 30 15:17:17.598062 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.598035 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/12fbfdca-4ab3-4f52-84aa-430978a4ee68-manager-config\") pod \"lws-controller-manager-556f467f75-szpt6\" (UID: \"12fbfdca-4ab3-4f52-84aa-430978a4ee68\") " pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" Jul 30 15:17:17.598173 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.598092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-876jl\" (UniqueName: \"kubernetes.io/projected/12fbfdca-4ab3-4f52-84aa-430978a4ee68-kube-api-access-876jl\") pod \"lws-controller-manager-556f467f75-szpt6\" (UID: \"12fbfdca-4ab3-4f52-84aa-430978a4ee68\") " pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" Jul 30 15:17:17.598173 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.598120 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12fbfdca-4ab3-4f52-84aa-430978a4ee68-cert\") pod \"lws-controller-manager-556f467f75-szpt6\" (UID: \"12fbfdca-4ab3-4f52-84aa-430978a4ee68\") " pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" Jul 30 15:17:17.598252 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.598227 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/12fbfdca-4ab3-4f52-84aa-430978a4ee68-metrics-cert\") pod \"lws-controller-manager-556f467f75-szpt6\" (UID: \"12fbfdca-4ab3-4f52-84aa-430978a4ee68\") " pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" Jul 30 15:17:17.699033 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.698949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12fbfdca-4ab3-4f52-84aa-430978a4ee68-cert\") pod \"lws-controller-manager-556f467f75-szpt6\" (UID: \"12fbfdca-4ab3-4f52-84aa-430978a4ee68\") " pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" Jul 30 15:17:17.699033 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.699002 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3fe5e6e-f812-4339-9345-481b4ee110c7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.699033 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.699032 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3fe5e6e-f812-4339-9345-481b4ee110c7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.699291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.699055 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn2n8\" (UniqueName: \"kubernetes.io/projected/b3fe5e6e-f812-4339-9345-481b4ee110c7-kube-api-access-nn2n8\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.699291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.699176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3fe5e6e-f812-4339-9345-481b4ee110c7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.699291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.699229 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3fe5e6e-f812-4339-9345-481b4ee110c7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.699291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.699274 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/12fbfdca-4ab3-4f52-84aa-430978a4ee68-metrics-cert\") pod \"lws-controller-manager-556f467f75-szpt6\" (UID: \"12fbfdca-4ab3-4f52-84aa-430978a4ee68\") " pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" Jul 30 15:17:17.699479 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.699453 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3fe5e6e-f812-4339-9345-481b4ee110c7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.699539 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.699508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3fe5e6e-f812-4339-9345-481b4ee110c7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.699590 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.699539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3fe5e6e-f812-4339-9345-481b4ee110c7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.699643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.699602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/12fbfdca-4ab3-4f52-84aa-430978a4ee68-manager-config\") pod \"lws-controller-manager-556f467f75-szpt6\" (UID: \"12fbfdca-4ab3-4f52-84aa-430978a4ee68\") " pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" Jul 30 15:17:17.699780 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.699762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-876jl\" (UniqueName: \"kubernetes.io/projected/12fbfdca-4ab3-4f52-84aa-430978a4ee68-kube-api-access-876jl\") pod \"lws-controller-manager-556f467f75-szpt6\" (UID: \"12fbfdca-4ab3-4f52-84aa-430978a4ee68\") " pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" Jul 30 15:17:17.699949 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.699930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3fe5e6e-f812-4339-9345-481b4ee110c7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.700457 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.700410 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/12fbfdca-4ab3-4f52-84aa-430978a4ee68-manager-config\") pod \"lws-controller-manager-556f467f75-szpt6\" (UID: \"12fbfdca-4ab3-4f52-84aa-430978a4ee68\") " pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" Jul 30 15:17:17.701847 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.701825 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12fbfdca-4ab3-4f52-84aa-430978a4ee68-cert\") pod \"lws-controller-manager-556f467f75-szpt6\" (UID: \"12fbfdca-4ab3-4f52-84aa-430978a4ee68\") " pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" Jul 30 15:17:17.702254 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.702233 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/12fbfdca-4ab3-4f52-84aa-430978a4ee68-metrics-cert\") pod \"lws-controller-manager-556f467f75-szpt6\" (UID: \"12fbfdca-4ab3-4f52-84aa-430978a4ee68\") " pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" Jul 30 15:17:17.722201 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.722170 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-876jl\" (UniqueName: \"kubernetes.io/projected/12fbfdca-4ab3-4f52-84aa-430978a4ee68-kube-api-access-876jl\") pod \"lws-controller-manager-556f467f75-szpt6\" (UID: \"12fbfdca-4ab3-4f52-84aa-430978a4ee68\") " pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" Jul 30 15:17:17.759273 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.758965 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" Jul 30 15:17:17.801604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.800938 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3fe5e6e-f812-4339-9345-481b4ee110c7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.801604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.800983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3fe5e6e-f812-4339-9345-481b4ee110c7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.801604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.801014 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn2n8\" (UniqueName: \"kubernetes.io/projected/b3fe5e6e-f812-4339-9345-481b4ee110c7-kube-api-access-nn2n8\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.801604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.801088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3fe5e6e-f812-4339-9345-481b4ee110c7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.801604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.801118 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3fe5e6e-f812-4339-9345-481b4ee110c7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.801604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.801183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3fe5e6e-f812-4339-9345-481b4ee110c7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.801604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.801215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3fe5e6e-f812-4339-9345-481b4ee110c7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.801604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.801244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3fe5e6e-f812-4339-9345-481b4ee110c7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.801604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.801249 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3fe5e6e-f812-4339-9345-481b4ee110c7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.801604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.801315 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3fe5e6e-f812-4339-9345-481b4ee110c7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.801604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.801577 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3fe5e6e-f812-4339-9345-481b4ee110c7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.802335 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.801899 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3fe5e6e-f812-4339-9345-481b4ee110c7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.802335 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.802106 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3fe5e6e-f812-4339-9345-481b4ee110c7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.802444 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.802388 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3fe5e6e-f812-4339-9345-481b4ee110c7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.804539 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.804514 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3fe5e6e-f812-4339-9345-481b4ee110c7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.805127 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.805109 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3fe5e6e-f812-4339-9345-481b4ee110c7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.816021 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.815994 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3fe5e6e-f812-4339-9345-481b4ee110c7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.818438 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.818394 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn2n8\" (UniqueName: \"kubernetes.io/projected/b3fe5e6e-f812-4339-9345-481b4ee110c7-kube-api-access-nn2n8\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf\" (UID: \"b3fe5e6e-f812-4339-9345-481b4ee110c7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.820758 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.820737 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:17.929505 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.929467 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-556f467f75-szpt6"] Jul 30 15:17:17.931636 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:17:17.931602 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12fbfdca_4ab3_4f52_84aa_430978a4ee68.slice/crio-e95bfffe97e449ddd38e87cdcd09920b26e728c6f9e0eff5a15fcdf924f8d613 WatchSource:0}: Error finding container e95bfffe97e449ddd38e87cdcd09920b26e728c6f9e0eff5a15fcdf924f8d613: Status 404 returned error can't find the container with id e95bfffe97e449ddd38e87cdcd09920b26e728c6f9e0eff5a15fcdf924f8d613 Jul 30 15:17:17.978587 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:17.978550 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf"] Jul 30 15:17:17.980720 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:17:17.980694 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3fe5e6e_f812_4339_9345_481b4ee110c7.slice/crio-1dde67040af5fe35104b044920856e77006fefefe1ad1d006ff62d5ee17c776a WatchSource:0}: Error finding container 1dde67040af5fe35104b044920856e77006fefefe1ad1d006ff62d5ee17c776a: Status 404 returned error can't find the container with id 1dde67040af5fe35104b044920856e77006fefefe1ad1d006ff62d5ee17c776a Jul 30 15:17:18.024033 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:18.024007 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" event={"ID":"12fbfdca-4ab3-4f52-84aa-430978a4ee68","Type":"ContainerStarted","Data":"e95bfffe97e449ddd38e87cdcd09920b26e728c6f9e0eff5a15fcdf924f8d613"} Jul 30 15:17:18.025164 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:18.025137 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" event={"ID":"b3fe5e6e-f812-4339-9345-481b4ee110c7","Type":"ContainerStarted","Data":"1dde67040af5fe35104b044920856e77006fefefe1ad1d006ff62d5ee17c776a"} Jul 30 15:17:19.148985 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:19.148948 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d7jmv"] Jul 30 15:17:19.153222 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:19.153147 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7jmv" Jul 30 15:17:19.157287 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:19.157258 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 30 15:17:19.163348 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:19.162381 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-fxhr7\"" Jul 30 15:17:19.186166 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:19.186121 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d7jmv"] Jul 30 15:17:19.318823 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:19.318000 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldj9m\" (UniqueName: \"kubernetes.io/projected/344cb966-68b6-4cb1-8206-3d822aece19b-kube-api-access-ldj9m\") pod \"kserve-controller-manager-d6f4b65b8-d7jmv\" (UID: \"344cb966-68b6-4cb1-8206-3d822aece19b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7jmv" Jul 30 15:17:19.318823 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:19.318100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/344cb966-68b6-4cb1-8206-3d822aece19b-cert\") pod \"kserve-controller-manager-d6f4b65b8-d7jmv\" (UID: \"344cb966-68b6-4cb1-8206-3d822aece19b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7jmv" Jul 30 15:17:19.419504 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:19.419393 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldj9m\" (UniqueName: \"kubernetes.io/projected/344cb966-68b6-4cb1-8206-3d822aece19b-kube-api-access-ldj9m\") pod \"kserve-controller-manager-d6f4b65b8-d7jmv\" (UID: \"344cb966-68b6-4cb1-8206-3d822aece19b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7jmv" Jul 30 15:17:19.419504 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:19.419473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/344cb966-68b6-4cb1-8206-3d822aece19b-cert\") pod \"kserve-controller-manager-d6f4b65b8-d7jmv\" (UID: \"344cb966-68b6-4cb1-8206-3d822aece19b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7jmv" Jul 30 15:17:19.424720 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:19.424669 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/344cb966-68b6-4cb1-8206-3d822aece19b-cert\") pod \"kserve-controller-manager-d6f4b65b8-d7jmv\" (UID: \"344cb966-68b6-4cb1-8206-3d822aece19b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7jmv" Jul 30 15:17:19.439594 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:19.439540 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldj9m\" (UniqueName: \"kubernetes.io/projected/344cb966-68b6-4cb1-8206-3d822aece19b-kube-api-access-ldj9m\") pod \"kserve-controller-manager-d6f4b65b8-d7jmv\" (UID: \"344cb966-68b6-4cb1-8206-3d822aece19b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7jmv" Jul 30 15:17:19.469424 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:19.469207 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7jmv" Jul 30 15:17:19.892367 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:19.892333 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d7jmv"] Jul 30 15:17:19.896063 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:17:19.896029 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344cb966_68b6_4cb1_8206_3d822aece19b.slice/crio-fd88d5cd5764cf6d1533c8a301a5b24c304d4a50017c269eb8d20e788f22fe8c WatchSource:0}: Error finding container fd88d5cd5764cf6d1533c8a301a5b24c304d4a50017c269eb8d20e788f22fe8c: Status 404 returned error can't find the container with id fd88d5cd5764cf6d1533c8a301a5b24c304d4a50017c269eb8d20e788f22fe8c Jul 30 15:17:20.043593 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:20.043476 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7jmv" event={"ID":"344cb966-68b6-4cb1-8206-3d822aece19b","Type":"ContainerStarted","Data":"fd88d5cd5764cf6d1533c8a301a5b24c304d4a50017c269eb8d20e788f22fe8c"} Jul 30 15:17:22.266158 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:22.266111 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:17:22.266438 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:22.266202 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:17:22.266438 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:22.266233 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:17:23.059656 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:23.059612 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" event={"ID":"12fbfdca-4ab3-4f52-84aa-430978a4ee68","Type":"ContainerStarted","Data":"5d0292b14ebbdfe9104d1cec14ed9b8bfc45355af39f49fd63d7a12c848e79bb"} Jul 30 15:17:23.059908 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:23.059708 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" Jul 30 15:17:23.061119 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:23.061095 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" event={"ID":"b3fe5e6e-f812-4339-9345-481b4ee110c7","Type":"ContainerStarted","Data":"9060f24add6b316712cfd1702df0c240e44783b55c663c508b722f776530a410"} Jul 30 15:17:23.109739 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:23.109689 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" podStartSLOduration=1.7808615319999999 podStartE2EDuration="6.109674483s" totalImagesPullingTime="4.328812951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:17:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:17:23.096852636 +0000 UTC m=+470.331983484" watchObservedRunningTime="2026-07-30 15:17:23.109674483 +0000 UTC m=+470.344805329" Jul 30 15:17:23.138710 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:23.138649 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" podStartSLOduration=1.8554368399999999 podStartE2EDuration="6.13863364s" totalImagesPullingTime="4.2831968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:17:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:17:23.137197811 +0000 UTC m=+470.372328656" watchObservedRunningTime="2026-07-30 15:17:23.13863364 +0000 UTC m=+470.373764486" Jul 30 15:17:23.821218 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:23.821181 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:23.826829 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:23.826784 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:24.067072 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:24.067026 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:24.068078 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:24.068052 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf" Jul 30 15:17:25.072447 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:25.072410 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7jmv" event={"ID":"344cb966-68b6-4cb1-8206-3d822aece19b","Type":"ContainerStarted","Data":"38f97a424d9bb828d9feaee0133078416be892afe6d79506bbe744b22eaa28cf"} Jul 30 15:17:25.072846 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:25.072722 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7jmv" Jul 30 15:17:25.108361 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:25.108307 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7jmv" podStartSLOduration=1.4380422259999999 podStartE2EDuration="6.108293812s" totalImagesPullingTime="4.670251586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:17:25.106570228 +0000 UTC m=+472.341701076" watchObservedRunningTime="2026-07-30 15:17:25.108293812 +0000 UTC m=+472.343424658" Jul 30 15:17:34.069386 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:34.069356 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-556f467f75-szpt6" Jul 30 15:17:56.082858 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:17:56.082827 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7jmv" Jul 30 15:18:53.622969 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:18:53.622890 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-s8vct"] Jul 30 15:18:53.625615 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:18:53.625598 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s8vct" Jul 30 15:18:53.628652 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:18:53.628629 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 30 15:18:53.633771 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:18:53.633748 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 30 15:18:53.633991 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:18:53.633976 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 30 15:18:53.634037 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:18:53.634024 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-jrs7r\"" Jul 30 15:18:53.649244 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:18:53.649220 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-s8vct"] Jul 30 15:18:53.706474 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:18:53.706444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftfzb\" (UniqueName: \"kubernetes.io/projected/e3b1403e-2d47-48fe-9195-330a8aeaa09e-kube-api-access-ftfzb\") pod \"dns-operator-controller-manager-864dd78c4c-s8vct\" (UID: \"e3b1403e-2d47-48fe-9195-330a8aeaa09e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s8vct" Jul 30 15:18:53.806909 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:18:53.806883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftfzb\" (UniqueName: \"kubernetes.io/projected/e3b1403e-2d47-48fe-9195-330a8aeaa09e-kube-api-access-ftfzb\") pod \"dns-operator-controller-manager-864dd78c4c-s8vct\" (UID: \"e3b1403e-2d47-48fe-9195-330a8aeaa09e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s8vct" Jul 30 15:18:53.817926 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:18:53.817891 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftfzb\" (UniqueName: \"kubernetes.io/projected/e3b1403e-2d47-48fe-9195-330a8aeaa09e-kube-api-access-ftfzb\") pod \"dns-operator-controller-manager-864dd78c4c-s8vct\" (UID: \"e3b1403e-2d47-48fe-9195-330a8aeaa09e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s8vct" Jul 30 15:18:53.935758 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:18:53.935679 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s8vct" Jul 30 15:18:54.060363 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:18:54.060338 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-s8vct"] Jul 30 15:18:54.062135 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:18:54.062107 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b1403e_2d47_48fe_9195_330a8aeaa09e.slice/crio-05a1b708f272cfa7107d5d16c80ae661bf2b2a687ec2dc82f2afa6b04b7c839c WatchSource:0}: Error finding container 05a1b708f272cfa7107d5d16c80ae661bf2b2a687ec2dc82f2afa6b04b7c839c: Status 404 returned error can't find the container with id 05a1b708f272cfa7107d5d16c80ae661bf2b2a687ec2dc82f2afa6b04b7c839c Jul 30 15:18:54.459377 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:18:54.459346 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s8vct" event={"ID":"e3b1403e-2d47-48fe-9195-330a8aeaa09e","Type":"ContainerStarted","Data":"05a1b708f272cfa7107d5d16c80ae661bf2b2a687ec2dc82f2afa6b04b7c839c"} Jul 30 15:18:56.470987 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:18:56.470940 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s8vct" event={"ID":"e3b1403e-2d47-48fe-9195-330a8aeaa09e","Type":"ContainerStarted","Data":"9f9340ba31288e4fdf6f8f99a86d2f881e0d009a9458f463bfc07c80778d7fa6"} Jul 30 15:18:56.471397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:18:56.471022 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s8vct" Jul 30 15:18:56.492026 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:18:56.491982 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s8vct" podStartSLOduration=1.225344578 podStartE2EDuration="3.491967416s" totalImagesPullingTime="2.266622838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:18:56.489665447 +0000 UTC m=+563.724796294" watchObservedRunningTime="2026-07-30 15:18:56.491967416 +0000 UTC m=+563.727098261" Jul 30 15:19:07.477598 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:07.477568 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s8vct" Jul 30 15:19:09.206544 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:09.206510 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bqchp"] Jul 30 15:19:09.209306 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:09.209284 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bqchp" Jul 30 15:19:09.212416 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:09.212392 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5k5zk\"" Jul 30 15:19:09.225134 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:09.225108 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bqchp"] Jul 30 15:19:09.233951 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:09.233929 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnbd8\" (UniqueName: \"kubernetes.io/projected/a6c52501-2f20-40d2-8541-76d6285e51bc-kube-api-access-gnbd8\") pod \"limitador-operator-controller-manager-859d7b78bd-bqchp\" (UID: \"a6c52501-2f20-40d2-8541-76d6285e51bc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bqchp" Jul 30 15:19:09.334898 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:09.334870 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnbd8\" (UniqueName: \"kubernetes.io/projected/a6c52501-2f20-40d2-8541-76d6285e51bc-kube-api-access-gnbd8\") pod \"limitador-operator-controller-manager-859d7b78bd-bqchp\" (UID: \"a6c52501-2f20-40d2-8541-76d6285e51bc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bqchp" Jul 30 15:19:09.343282 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:09.343263 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnbd8\" (UniqueName: \"kubernetes.io/projected/a6c52501-2f20-40d2-8541-76d6285e51bc-kube-api-access-gnbd8\") pod \"limitador-operator-controller-manager-859d7b78bd-bqchp\" (UID: \"a6c52501-2f20-40d2-8541-76d6285e51bc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bqchp" Jul 30 15:19:09.520462 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:09.520380 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bqchp" Jul 30 15:19:09.652543 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:09.652519 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bqchp"] Jul 30 15:19:09.654527 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:19:09.654499 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c52501_2f20_40d2_8541_76d6285e51bc.slice/crio-fd6d4fc165829d21cc951022aa565b39a601b8e8a78babacc83dcf91dda040d4 WatchSource:0}: Error finding container fd6d4fc165829d21cc951022aa565b39a601b8e8a78babacc83dcf91dda040d4: Status 404 returned error can't find the container with id fd6d4fc165829d21cc951022aa565b39a601b8e8a78babacc83dcf91dda040d4 Jul 30 15:19:10.533933 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:10.533899 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bqchp" event={"ID":"a6c52501-2f20-40d2-8541-76d6285e51bc","Type":"ContainerStarted","Data":"fd6d4fc165829d21cc951022aa565b39a601b8e8a78babacc83dcf91dda040d4"} Jul 30 15:19:11.539879 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:11.539838 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bqchp" event={"ID":"a6c52501-2f20-40d2-8541-76d6285e51bc","Type":"ContainerStarted","Data":"28dcbffd14071bf163303ccf8f54d1ca03fe53ccf128327a17b87b8e326df8fc"} Jul 30 15:19:11.540269 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:11.539914 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bqchp" Jul 30 15:19:11.560005 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:11.559959 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bqchp" podStartSLOduration=0.829734258 podStartE2EDuration="2.559943042s" totalImagesPullingTime="1.730208784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:19:11.558648651 +0000 UTC m=+578.793779498" watchObservedRunningTime="2026-07-30 15:19:11.559943042 +0000 UTC m=+578.795073916" Jul 30 15:19:22.546045 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:22.546015 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bqchp" Jul 30 15:19:49.923681 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:49.923651 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh"] Jul 30 15:19:49.932038 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:49.932016 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:49.935007 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:49.934981 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-n47qd\"" Jul 30 15:19:49.940434 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:49.940408 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh"] Jul 30 15:19:50.072543 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.072508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2915a001-9525-405e-a5cb-397ad774e66a-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.072710 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.072565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2915a001-9525-405e-a5cb-397ad774e66a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.072710 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.072586 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2915a001-9525-405e-a5cb-397ad774e66a-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.072710 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.072610 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2915a001-9525-405e-a5cb-397ad774e66a-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.072710 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.072635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwcjh\" (UniqueName: \"kubernetes.io/projected/2915a001-9525-405e-a5cb-397ad774e66a-kube-api-access-pwcjh\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.072880 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.072717 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2915a001-9525-405e-a5cb-397ad774e66a-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.072880 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.072755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2915a001-9525-405e-a5cb-397ad774e66a-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.072880 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.072779 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2915a001-9525-405e-a5cb-397ad774e66a-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.072880 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.072840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2915a001-9525-405e-a5cb-397ad774e66a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.174059 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.173983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwcjh\" (UniqueName: \"kubernetes.io/projected/2915a001-9525-405e-a5cb-397ad774e66a-kube-api-access-pwcjh\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.174059 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.174042 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2915a001-9525-405e-a5cb-397ad774e66a-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.174281 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.174072 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2915a001-9525-405e-a5cb-397ad774e66a-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.174281 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.174097 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2915a001-9525-405e-a5cb-397ad774e66a-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.174281 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.174127 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2915a001-9525-405e-a5cb-397ad774e66a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.174281 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.174262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2915a001-9525-405e-a5cb-397ad774e66a-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.174479 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.174352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2915a001-9525-405e-a5cb-397ad774e66a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.174479 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.174385 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2915a001-9525-405e-a5cb-397ad774e66a-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.174479 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.174411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2915a001-9525-405e-a5cb-397ad774e66a-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.174635 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.174611 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2915a001-9525-405e-a5cb-397ad774e66a-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.174695 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.174652 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2915a001-9525-405e-a5cb-397ad774e66a-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.174775 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.174752 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2915a001-9525-405e-a5cb-397ad774e66a-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.174918 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.174900 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2915a001-9525-405e-a5cb-397ad774e66a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.175018 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.175001 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2915a001-9525-405e-a5cb-397ad774e66a-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.176314 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.176291 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2915a001-9525-405e-a5cb-397ad774e66a-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.176639 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.176623 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2915a001-9525-405e-a5cb-397ad774e66a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.183239 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.183216 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwcjh\" (UniqueName: \"kubernetes.io/projected/2915a001-9525-405e-a5cb-397ad774e66a-kube-api-access-pwcjh\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.183317 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.183251 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2915a001-9525-405e-a5cb-397ad774e66a-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gkwnh\" (UID: \"2915a001-9525-405e-a5cb-397ad774e66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.244001 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.243980 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:50.373442 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.373407 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh"] Jul 30 15:19:50.376822 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:19:50.376781 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2915a001_9525_405e_a5cb_397ad774e66a.slice/crio-77db967dbfa99b596c42057a9fd5f469c3cba0b359db400dd78d7e23678b2c7f WatchSource:0}: Error finding container 77db967dbfa99b596c42057a9fd5f469c3cba0b359db400dd78d7e23678b2c7f: Status 404 returned error can't find the container with id 77db967dbfa99b596c42057a9fd5f469c3cba0b359db400dd78d7e23678b2c7f Jul 30 15:19:50.379365 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.379324 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:19:50.379459 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.379398 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:19:50.379459 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.379431 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:19:50.714470 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.714390 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" event={"ID":"2915a001-9525-405e-a5cb-397ad774e66a","Type":"ContainerStarted","Data":"a9313ad4fd7d42d6d869a981b111b843d0ab8c1baaff3c0ce3744b970b9d9d93"} Jul 30 15:19:50.714470 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.714452 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" event={"ID":"2915a001-9525-405e-a5cb-397ad774e66a","Type":"ContainerStarted","Data":"77db967dbfa99b596c42057a9fd5f469c3cba0b359db400dd78d7e23678b2c7f"} Jul 30 15:19:50.739827 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:50.739751 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" podStartSLOduration=1.739737072 podStartE2EDuration="1.739737072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:19:50.737155849 +0000 UTC m=+617.972286696" watchObservedRunningTime="2026-07-30 15:19:50.739737072 +0000 UTC m=+617.974867917" Jul 30 15:19:51.245048 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:51.245013 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:51.249669 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:51.249645 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:51.719369 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:51.719333 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:19:51.720369 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:19:51.720350 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:20:02.826329 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:02.826238 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:20:02.830048 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:02.830025 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-tjvjw" Jul 30 15:20:02.833014 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:02.832990 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 30 15:20:02.833132 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:02.833026 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vzl9h\"" Jul 30 15:20:02.838965 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:02.838942 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:20:02.871753 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:02.871715 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:20:02.881236 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:02.881210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ae3de39f-55f2-4484-ba76-0f6a11077a1c-config-file\") pod \"limitador-limitador-54d6d988d5-tjvjw\" (UID: \"ae3de39f-55f2-4484-ba76-0f6a11077a1c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tjvjw" Jul 30 15:20:02.881354 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:02.881239 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6czgv\" (UniqueName: \"kubernetes.io/projected/ae3de39f-55f2-4484-ba76-0f6a11077a1c-kube-api-access-6czgv\") pod \"limitador-limitador-54d6d988d5-tjvjw\" (UID: \"ae3de39f-55f2-4484-ba76-0f6a11077a1c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tjvjw" Jul 30 15:20:02.981833 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:02.981761 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ae3de39f-55f2-4484-ba76-0f6a11077a1c-config-file\") pod \"limitador-limitador-54d6d988d5-tjvjw\" (UID: \"ae3de39f-55f2-4484-ba76-0f6a11077a1c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tjvjw" Jul 30 15:20:02.982004 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:02.981837 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6czgv\" (UniqueName: \"kubernetes.io/projected/ae3de39f-55f2-4484-ba76-0f6a11077a1c-kube-api-access-6czgv\") pod \"limitador-limitador-54d6d988d5-tjvjw\" (UID: \"ae3de39f-55f2-4484-ba76-0f6a11077a1c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tjvjw" Jul 30 15:20:02.982396 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:02.982375 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ae3de39f-55f2-4484-ba76-0f6a11077a1c-config-file\") pod \"limitador-limitador-54d6d988d5-tjvjw\" (UID: \"ae3de39f-55f2-4484-ba76-0f6a11077a1c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tjvjw" Jul 30 15:20:02.990372 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:02.990343 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6czgv\" (UniqueName: \"kubernetes.io/projected/ae3de39f-55f2-4484-ba76-0f6a11077a1c-kube-api-access-6czgv\") pod \"limitador-limitador-54d6d988d5-tjvjw\" (UID: \"ae3de39f-55f2-4484-ba76-0f6a11077a1c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tjvjw" Jul 30 15:20:03.142316 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:03.142281 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-tjvjw" Jul 30 15:20:03.279466 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:03.279437 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:20:03.281289 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:20:03.281259 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3de39f_55f2_4484_ba76_0f6a11077a1c.slice/crio-0d49d9f59900a4f2c24a30ec9541bd4b858860239cd4573b3c885664459e3090 WatchSource:0}: Error finding container 0d49d9f59900a4f2c24a30ec9541bd4b858860239cd4573b3c885664459e3090: Status 404 returned error can't find the container with id 0d49d9f59900a4f2c24a30ec9541bd4b858860239cd4573b3c885664459e3090 Jul 30 15:20:03.770896 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:03.770861 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-tjvjw" event={"ID":"ae3de39f-55f2-4484-ba76-0f6a11077a1c","Type":"ContainerStarted","Data":"0d49d9f59900a4f2c24a30ec9541bd4b858860239cd4573b3c885664459e3090"} Jul 30 15:20:06.787372 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:06.787336 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-tjvjw" event={"ID":"ae3de39f-55f2-4484-ba76-0f6a11077a1c","Type":"ContainerStarted","Data":"01941165c30eb056160f6c986ec6adb1d64a36b861b070904c85e962ad3496d8"} Jul 30 15:20:06.787775 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:06.787471 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-tjvjw" Jul 30 15:20:06.806871 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:06.806818 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-tjvjw" podStartSLOduration=1.78207155 podStartE2EDuration="4.806788995s" totalImagesPullingTime="3.024717445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:20:06.804919035 +0000 UTC m=+634.040049883" watchObservedRunningTime="2026-07-30 15:20:06.806788995 +0000 UTC m=+634.041919889" Jul 30 15:20:17.792546 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:20:17.792516 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-tjvjw" Jul 30 15:21:23.909117 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:23.909081 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:21:32.188993 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:32.188959 2579 generic.go:356] "Generic (PLEG): container finished" podID="2915a001-9525-405e-a5cb-397ad774e66a" containerID="a9313ad4fd7d42d6d869a981b111b843d0ab8c1baaff3c0ce3744b970b9d9d93" exitCode=0 Jul 30 15:21:32.189405 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:32.189031 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" event={"ID":"2915a001-9525-405e-a5cb-397ad774e66a","Type":"ContainerDied","Data":"a9313ad4fd7d42d6d869a981b111b843d0ab8c1baaff3c0ce3744b970b9d9d93"} Jul 30 15:21:32.189405 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:32.189399 2579 scope.go:117] "RemoveContainer" containerID="a9313ad4fd7d42d6d869a981b111b843d0ab8c1baaff3c0ce3744b970b9d9d93" Jul 30 15:21:32.190398 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:32.190365 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:21:32.190472 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:32.190456 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:21:32.190522 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:32.190506 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:21:33.205187 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:33.205142 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" event={"ID":"2915a001-9525-405e-a5cb-397ad774e66a","Type":"ContainerStarted","Data":"2c64f4d0151685b0b89754defffb2c5b0f785dbab599ae2d281fd649e8b9cd70"} Jul 30 15:21:33.244772 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:33.244740 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:21:33.245821 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:33.245764 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-gkwnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Jul 30 15:21:33.245959 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:33.245836 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" podUID="2915a001-9525-405e-a5cb-397ad774e66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Jul 30 15:21:34.244711 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:34.244673 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-gkwnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Jul 30 15:21:34.245117 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:34.244740 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" podUID="2915a001-9525-405e-a5cb-397ad774e66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Jul 30 15:21:35.244496 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:35.244465 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-gkwnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Jul 30 15:21:35.244667 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:35.244522 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" podUID="2915a001-9525-405e-a5cb-397ad774e66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Jul 30 15:21:36.244663 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:36.244628 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-gkwnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Jul 30 15:21:36.245070 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:36.244687 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" podUID="2915a001-9525-405e-a5cb-397ad774e66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Jul 30 15:21:36.719781 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:36.719733 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:21:37.248816 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:37.248769 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:21:37.250257 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:37.250235 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gkwnh" Jul 30 15:21:50.049639 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:21:50.049604 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:22:01.926520 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:01.926481 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq"] Jul 30 15:22:01.929619 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:01.929599 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:01.939871 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:01.939850 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 30 15:22:01.939985 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:01.939891 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 30 15:22:01.939985 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:01.939955 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 30 15:22:01.940072 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:01.940009 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tp2xj\"" Jul 30 15:22:01.954873 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:01.954843 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq"] Jul 30 15:22:02.010961 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.010929 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf6af2f-2130-4af2-8790-d17c6a2eb6c1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xhxq\" (UID: \"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.010961 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.010963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf6af2f-2130-4af2-8790-d17c6a2eb6c1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xhxq\" (UID: \"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.011176 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.010994 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bqtm\" (UniqueName: \"kubernetes.io/projected/bbf6af2f-2130-4af2-8790-d17c6a2eb6c1-kube-api-access-2bqtm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xhxq\" (UID: \"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.011176 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.011074 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf6af2f-2130-4af2-8790-d17c6a2eb6c1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xhxq\" (UID: \"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.011176 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.011108 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf6af2f-2130-4af2-8790-d17c6a2eb6c1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xhxq\" (UID: \"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.011176 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.011124 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf6af2f-2130-4af2-8790-d17c6a2eb6c1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xhxq\" (UID: \"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.112326 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.112289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf6af2f-2130-4af2-8790-d17c6a2eb6c1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xhxq\" (UID: \"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.112326 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.112329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf6af2f-2130-4af2-8790-d17c6a2eb6c1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xhxq\" (UID: \"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.112533 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.112364 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bqtm\" (UniqueName: \"kubernetes.io/projected/bbf6af2f-2130-4af2-8790-d17c6a2eb6c1-kube-api-access-2bqtm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xhxq\" (UID: \"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.112533 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.112400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf6af2f-2130-4af2-8790-d17c6a2eb6c1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xhxq\" (UID: \"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.112533 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.112419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf6af2f-2130-4af2-8790-d17c6a2eb6c1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xhxq\" (UID: \"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.112533 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.112436 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf6af2f-2130-4af2-8790-d17c6a2eb6c1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xhxq\" (UID: \"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.112758 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.112734 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf6af2f-2130-4af2-8790-d17c6a2eb6c1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xhxq\" (UID: \"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.112842 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.112772 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf6af2f-2130-4af2-8790-d17c6a2eb6c1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xhxq\" (UID: \"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.112897 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.112849 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf6af2f-2130-4af2-8790-d17c6a2eb6c1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xhxq\" (UID: \"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.114687 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.114670 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf6af2f-2130-4af2-8790-d17c6a2eb6c1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xhxq\" (UID: \"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.114973 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.114958 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf6af2f-2130-4af2-8790-d17c6a2eb6c1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xhxq\" (UID: \"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.127445 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.127422 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bqtm\" (UniqueName: \"kubernetes.io/projected/bbf6af2f-2130-4af2-8790-d17c6a2eb6c1-kube-api-access-2bqtm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xhxq\" (UID: \"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.239658 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.239575 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:02.374984 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.374961 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq"] Jul 30 15:22:02.376750 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:22:02.376726 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf6af2f_2130_4af2_8790_d17c6a2eb6c1.slice/crio-df2e9ff15805e4e4c76fbd17e1abd668a3306728c5b56bda3f719b5bacb9f12c WatchSource:0}: Error finding container df2e9ff15805e4e4c76fbd17e1abd668a3306728c5b56bda3f719b5bacb9f12c: Status 404 returned error can't find the container with id df2e9ff15805e4e4c76fbd17e1abd668a3306728c5b56bda3f719b5bacb9f12c Jul 30 15:22:02.378045 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.378027 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 15:22:02.936427 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:02.936388 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:22:03.344374 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:03.344285 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" event={"ID":"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1","Type":"ContainerStarted","Data":"df2e9ff15805e4e4c76fbd17e1abd668a3306728c5b56bda3f719b5bacb9f12c"} Jul 30 15:22:04.350191 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:04.350158 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" event={"ID":"bbf6af2f-2130-4af2-8790-d17c6a2eb6c1","Type":"ContainerStarted","Data":"5e895d82edf81907bab6fbb2a79496201e6b23171eefc0621450e27d57932e3c"} Jul 30 15:22:04.372009 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:04.371955 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" podStartSLOduration=2.24538583 podStartE2EDuration="3.371938919s" totalImagesPullingTime="1.126553089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:22:04.371264947 +0000 UTC m=+751.606395791" watchObservedRunningTime="2026-07-30 15:22:04.371938919 +0000 UTC m=+751.607069765" Jul 30 15:22:07.755349 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:07.755312 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:22:12.239940 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:12.239857 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:12.239940 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:12.239916 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:12.251705 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:12.251680 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:12.396985 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:12.396955 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xhxq" Jul 30 15:22:25.240578 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.240541 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8"] Jul 30 15:22:25.244035 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.244011 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.246980 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.246954 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 30 15:22:25.254653 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.254627 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8"] Jul 30 15:22:25.407777 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.407743 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa820390-de40-4d51-904a-90c866d9917b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8\" (UID: \"fa820390-de40-4d51-904a-90c866d9917b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.407777 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.407777 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa820390-de40-4d51-904a-90c866d9917b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8\" (UID: \"fa820390-de40-4d51-904a-90c866d9917b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.408069 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.407827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa820390-de40-4d51-904a-90c866d9917b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8\" (UID: \"fa820390-de40-4d51-904a-90c866d9917b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.408069 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.407850 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa820390-de40-4d51-904a-90c866d9917b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8\" (UID: \"fa820390-de40-4d51-904a-90c866d9917b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.408069 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.407889 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa820390-de40-4d51-904a-90c866d9917b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8\" (UID: \"fa820390-de40-4d51-904a-90c866d9917b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.408069 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.407960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnkbq\" (UniqueName: \"kubernetes.io/projected/fa820390-de40-4d51-904a-90c866d9917b-kube-api-access-gnkbq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8\" (UID: \"fa820390-de40-4d51-904a-90c866d9917b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.508639 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.508569 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa820390-de40-4d51-904a-90c866d9917b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8\" (UID: \"fa820390-de40-4d51-904a-90c866d9917b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.508639 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.508606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa820390-de40-4d51-904a-90c866d9917b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8\" (UID: \"fa820390-de40-4d51-904a-90c866d9917b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.508639 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.508629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa820390-de40-4d51-904a-90c866d9917b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8\" (UID: \"fa820390-de40-4d51-904a-90c866d9917b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.508930 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.508647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa820390-de40-4d51-904a-90c866d9917b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8\" (UID: \"fa820390-de40-4d51-904a-90c866d9917b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.508930 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.508683 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa820390-de40-4d51-904a-90c866d9917b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8\" (UID: \"fa820390-de40-4d51-904a-90c866d9917b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.508930 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.508707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnkbq\" (UniqueName: \"kubernetes.io/projected/fa820390-de40-4d51-904a-90c866d9917b-kube-api-access-gnkbq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8\" (UID: \"fa820390-de40-4d51-904a-90c866d9917b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.509094 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.509052 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa820390-de40-4d51-904a-90c866d9917b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8\" (UID: \"fa820390-de40-4d51-904a-90c866d9917b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.509139 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.509107 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa820390-de40-4d51-904a-90c866d9917b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8\" (UID: \"fa820390-de40-4d51-904a-90c866d9917b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.509139 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.509126 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa820390-de40-4d51-904a-90c866d9917b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8\" (UID: \"fa820390-de40-4d51-904a-90c866d9917b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.510842 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.510826 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa820390-de40-4d51-904a-90c866d9917b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8\" (UID: \"fa820390-de40-4d51-904a-90c866d9917b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.510950 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.510933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa820390-de40-4d51-904a-90c866d9917b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8\" (UID: \"fa820390-de40-4d51-904a-90c866d9917b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.517876 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.517830 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnkbq\" (UniqueName: \"kubernetes.io/projected/fa820390-de40-4d51-904a-90c866d9917b-kube-api-access-gnkbq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8\" (UID: \"fa820390-de40-4d51-904a-90c866d9917b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.556008 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.555984 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:25.692624 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:25.692583 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8"] Jul 30 15:22:26.448485 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:26.448443 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" event={"ID":"fa820390-de40-4d51-904a-90c866d9917b","Type":"ContainerStarted","Data":"3ff796542e8754a442745ddfb7ebf60aa253993ab9379600fd52654356ab6691"} Jul 30 15:22:26.448916 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:26.448510 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" event={"ID":"fa820390-de40-4d51-904a-90c866d9917b","Type":"ContainerStarted","Data":"8fa8de81a02c1b2bf7c63bf5d848e68c1d892f12802907bbe9ec9730ac515493"} Jul 30 15:22:26.855546 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:26.855449 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" podStartSLOduration=1.6418860259999999 podStartE2EDuration="1.85543121s" totalImagesPullingTime="213.545184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:22:26.479729537 +0000 UTC m=+773.714860382" watchObservedRunningTime="2026-07-30 15:22:26.85543121 +0000 UTC m=+774.090562076" Jul 30 15:22:26.857094 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:26.857076 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:22:35.556295 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:35.556265 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:35.556676 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:35.556352 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:35.568707 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:35.568686 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:36.245780 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:36.245745 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:22:36.503834 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:36.503751 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8" Jul 30 15:22:45.336608 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:22:45.336574 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:24:24.346133 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:24:24.346099 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:24:34.533328 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:24:34.533253 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:24:44.239259 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:24:44.239224 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:26:21.038116 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:26:21.038078 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:26:33.435655 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:26:33.435621 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:27:47.144859 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:27:47.144768 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:27:55.134568 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:27:55.134526 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:28:10.258914 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:28:10.258877 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:28:18.173859 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:28:18.173828 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:28:30.570648 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:28:30.570612 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:28:41.950405 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:28:41.950368 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:28:53.242379 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:28:53.242339 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:29:03.637091 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:29:03.637014 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:29:24.934481 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:29:24.934448 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:29:48.435465 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:29:48.435421 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:30:09.444596 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:30:09.444560 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:30:20.678934 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:30:20.678900 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:30:37.849464 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:30:37.849371 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:30:57.831520 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:30:57.831486 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:31:13.038769 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:31:13.038727 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:31:22.842102 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:31:22.842057 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:31:33.340068 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:31:33.340036 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:31:45.837053 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:31:45.837019 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:31:55.540146 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:31:55.540063 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:32:06.541704 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:32:06.541672 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:32:28.537390 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:32:28.537336 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:34:53.832008 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:34:53.831975 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:35:02.438068 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:35:02.438026 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:35:27.444001 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:35:27.443966 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:35:35.837941 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:35:35.837904 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:35:49.148861 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:35:49.148826 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:35:57.544068 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:35:57.544034 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:36:09.743038 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:36:09.743003 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:36:27.948904 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:36:27.948827 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:36:39.936639 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:36:39.936603 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:36:51.536534 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:36:51.536501 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:37:00.134756 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:00.134713 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:37:23.744906 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:23.744873 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:37:31.356917 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:31.356884 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:37:54.241011 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.240977 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n"] Jul 30 15:37:54.244447 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.244430 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.247643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.247616 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-515950da-kserve-self-signed-certs\"" Jul 30 15:37:54.259984 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.259960 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n"] Jul 30 15:37:54.349085 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.349052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84575897-3456-4e6a-9469-312424133486-tls-certs\") pod \"e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.349267 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.349153 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2wvh\" (UniqueName: \"kubernetes.io/projected/84575897-3456-4e6a-9469-312424133486-kube-api-access-s2wvh\") pod \"e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.349267 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.349178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-dshm\") pod \"e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.349382 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.349285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-home\") pod \"e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.349382 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.349338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-model-cache\") pod \"e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.349499 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.349459 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-tmp-dir\") pod \"e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.450092 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.450057 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-tmp-dir\") pod \"e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.450092 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.450099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84575897-3456-4e6a-9469-312424133486-tls-certs\") pod \"e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.450351 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.450186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2wvh\" (UniqueName: \"kubernetes.io/projected/84575897-3456-4e6a-9469-312424133486-kube-api-access-s2wvh\") pod \"e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.450351 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.450216 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-dshm\") pod \"e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.450351 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.450250 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-home\") pod \"e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.450520 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.450398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-model-cache\") pod \"e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.450578 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.450532 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-home\") pod \"e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.450578 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.450549 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-tmp-dir\") pod \"e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.450756 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.450714 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-model-cache\") pod \"e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.452426 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.452402 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-dshm\") pod \"e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.452718 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.452698 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84575897-3456-4e6a-9469-312424133486-tls-certs\") pod \"e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.459400 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.459379 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2wvh\" (UniqueName: \"kubernetes.io/projected/84575897-3456-4e6a-9469-312424133486-kube-api-access-s2wvh\") pod \"e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.560359 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.560301 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:37:54.691026 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.691002 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n"] Jul 30 15:37:54.693120 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:37:54.693093 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84575897_3456_4e6a_9469_312424133486.slice/crio-d8f49331e7db436b2aa2ad21aaa7511ae69a3f02fb971a1462bd65453b70a75e WatchSource:0}: Error finding container d8f49331e7db436b2aa2ad21aaa7511ae69a3f02fb971a1462bd65453b70a75e: Status 404 returned error can't find the container with id d8f49331e7db436b2aa2ad21aaa7511ae69a3f02fb971a1462bd65453b70a75e Jul 30 15:37:54.800576 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.800537 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" event={"ID":"84575897-3456-4e6a-9469-312424133486","Type":"ContainerStarted","Data":"729fd032d9c197ee84539cf20c70157a4729b0c013213a1b32b270ddc495076e"} Jul 30 15:37:54.800732 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.800592 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" event={"ID":"84575897-3456-4e6a-9469-312424133486","Type":"ContainerStarted","Data":"d8f49331e7db436b2aa2ad21aaa7511ae69a3f02fb971a1462bd65453b70a75e"} Jul 30 15:37:54.821407 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:54.821310 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" podStartSLOduration=0.821294643 podStartE2EDuration="821.294643ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:37:54.819813637 +0000 UTC m=+1702.054944485" watchObservedRunningTime="2026-07-30 15:37:54.821294643 +0000 UTC m=+1702.056425490" Jul 30 15:37:57.642128 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:37:57.642048 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:38:04.560631 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:04.560545 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:38:04.560631 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:04.560599 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:38:04.572878 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:04.572853 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:38:04.862055 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:04.861969 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:38:07.431826 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.431778 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm"] Jul 30 15:38:07.442650 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.442621 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.447002 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.446975 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-515950da-kserve-self-signed-certs\"" Jul 30 15:38:07.447927 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.447901 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm"] Jul 30 15:38:07.571709 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.571681 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-tmp-dir\") pod \"e2e-same-id-b-515950da-kserve-6dbd695799-vnccm\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.571948 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.571758 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d39032-e166-46a5-afb7-fd2dd0bf4145-tls-certs\") pod \"e2e-same-id-b-515950da-kserve-6dbd695799-vnccm\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.571948 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.571789 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtdv9\" (UniqueName: \"kubernetes.io/projected/81d39032-e166-46a5-afb7-fd2dd0bf4145-kube-api-access-mtdv9\") pod \"e2e-same-id-b-515950da-kserve-6dbd695799-vnccm\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.571948 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.571836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-home\") pod \"e2e-same-id-b-515950da-kserve-6dbd695799-vnccm\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.571948 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.571862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-model-cache\") pod \"e2e-same-id-b-515950da-kserve-6dbd695799-vnccm\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.571948 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.571889 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-dshm\") pod \"e2e-same-id-b-515950da-kserve-6dbd695799-vnccm\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.672753 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.672724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d39032-e166-46a5-afb7-fd2dd0bf4145-tls-certs\") pod \"e2e-same-id-b-515950da-kserve-6dbd695799-vnccm\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.672938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.672761 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtdv9\" (UniqueName: \"kubernetes.io/projected/81d39032-e166-46a5-afb7-fd2dd0bf4145-kube-api-access-mtdv9\") pod \"e2e-same-id-b-515950da-kserve-6dbd695799-vnccm\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.672938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.672790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-home\") pod \"e2e-same-id-b-515950da-kserve-6dbd695799-vnccm\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.672938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.672853 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-model-cache\") pod \"e2e-same-id-b-515950da-kserve-6dbd695799-vnccm\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.672938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.672888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-dshm\") pod \"e2e-same-id-b-515950da-kserve-6dbd695799-vnccm\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.672938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.672925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-tmp-dir\") pod \"e2e-same-id-b-515950da-kserve-6dbd695799-vnccm\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.673192 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.673174 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-home\") pod \"e2e-same-id-b-515950da-kserve-6dbd695799-vnccm\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.673241 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.673215 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-model-cache\") pod \"e2e-same-id-b-515950da-kserve-6dbd695799-vnccm\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.673283 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.673256 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-tmp-dir\") pod \"e2e-same-id-b-515950da-kserve-6dbd695799-vnccm\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.675145 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.675126 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-dshm\") pod \"e2e-same-id-b-515950da-kserve-6dbd695799-vnccm\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.675413 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.675392 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d39032-e166-46a5-afb7-fd2dd0bf4145-tls-certs\") pod \"e2e-same-id-b-515950da-kserve-6dbd695799-vnccm\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.685088 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.685035 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtdv9\" (UniqueName: \"kubernetes.io/projected/81d39032-e166-46a5-afb7-fd2dd0bf4145-kube-api-access-mtdv9\") pod \"e2e-same-id-b-515950da-kserve-6dbd695799-vnccm\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.753174 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.753148 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:07.880301 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:07.880279 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm"] Jul 30 15:38:07.882943 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:38:07.882916 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81d39032_e166_46a5_afb7_fd2dd0bf4145.slice/crio-93dcd39b785af615944d8ce627aed389dfe4d7cf9353626d2cbfb0b168f4add8 WatchSource:0}: Error finding container 93dcd39b785af615944d8ce627aed389dfe4d7cf9353626d2cbfb0b168f4add8: Status 404 returned error can't find the container with id 93dcd39b785af615944d8ce627aed389dfe4d7cf9353626d2cbfb0b168f4add8 Jul 30 15:38:08.872267 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:08.872234 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" event={"ID":"81d39032-e166-46a5-afb7-fd2dd0bf4145","Type":"ContainerStarted","Data":"48e4f73d1d4da36a0025e20a93596f354e298fa030dfa88a0a3b9b6475a20540"} Jul 30 15:38:08.872660 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:08.872275 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" event={"ID":"81d39032-e166-46a5-afb7-fd2dd0bf4145","Type":"ContainerStarted","Data":"93dcd39b785af615944d8ce627aed389dfe4d7cf9353626d2cbfb0b168f4add8"} Jul 30 15:38:08.891301 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:08.891252 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" podStartSLOduration=1.891236377 podStartE2EDuration="1.891236377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:38:08.889544773 +0000 UTC m=+1716.124675622" watchObservedRunningTime="2026-07-30 15:38:08.891236377 +0000 UTC m=+1716.126367222" Jul 30 15:38:12.156587 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:12.156551 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:38:17.753741 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:17.753702 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:17.754210 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:17.753752 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:17.766395 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:17.766370 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:17.924832 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:17.924805 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:35.240337 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:35.240301 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n"] Jul 30 15:38:36.848065 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:36.848035 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm"] Jul 30 15:38:39.446919 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:39.446882 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:38:50.259872 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.259834 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" podUID="84575897-3456-4e6a-9469-312424133486" containerName="main" containerID="cri-o://729fd032d9c197ee84539cf20c70157a4729b0c013213a1b32b270ddc495076e" gracePeriod=45 Jul 30 15:38:50.513192 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.513134 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:38:50.558387 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.558356 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-model-cache\") pod \"84575897-3456-4e6a-9469-312424133486\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " Jul 30 15:38:50.558513 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.558427 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-home\") pod \"84575897-3456-4e6a-9469-312424133486\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " Jul 30 15:38:50.558513 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.558490 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-dshm\") pod \"84575897-3456-4e6a-9469-312424133486\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " Jul 30 15:38:50.558604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.558545 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-tmp-dir\") pod \"84575897-3456-4e6a-9469-312424133486\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " Jul 30 15:38:50.558604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.558581 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2wvh\" (UniqueName: \"kubernetes.io/projected/84575897-3456-4e6a-9469-312424133486-kube-api-access-s2wvh\") pod \"84575897-3456-4e6a-9469-312424133486\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " Jul 30 15:38:50.558724 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.558614 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84575897-3456-4e6a-9469-312424133486-tls-certs\") pod \"84575897-3456-4e6a-9469-312424133486\" (UID: \"84575897-3456-4e6a-9469-312424133486\") " Jul 30 15:38:50.558724 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.558627 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-model-cache" (OuterVolumeSpecName: "model-cache") pod "84575897-3456-4e6a-9469-312424133486" (UID: "84575897-3456-4e6a-9469-312424133486"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:38:50.558724 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.558645 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-home" (OuterVolumeSpecName: "home") pod "84575897-3456-4e6a-9469-312424133486" (UID: "84575897-3456-4e6a-9469-312424133486"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:38:50.558923 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.558809 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84575897-3456-4e6a-9469-312424133486" (UID: "84575897-3456-4e6a-9469-312424133486"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:38:50.559016 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.558995 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-model-cache\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:38:50.559075 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.559019 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-home\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:38:50.559075 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.559029 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-tmp-dir\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:38:50.560648 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.560620 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-dshm" (OuterVolumeSpecName: "dshm") pod "84575897-3456-4e6a-9469-312424133486" (UID: "84575897-3456-4e6a-9469-312424133486"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:38:50.560759 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.560644 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84575897-3456-4e6a-9469-312424133486-kube-api-access-s2wvh" (OuterVolumeSpecName: "kube-api-access-s2wvh") pod "84575897-3456-4e6a-9469-312424133486" (UID: "84575897-3456-4e6a-9469-312424133486"). InnerVolumeSpecName "kube-api-access-s2wvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:38:50.560759 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.560723 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84575897-3456-4e6a-9469-312424133486-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84575897-3456-4e6a-9469-312424133486" (UID: "84575897-3456-4e6a-9469-312424133486"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:38:50.659919 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.659893 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84575897-3456-4e6a-9469-312424133486-dshm\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:38:50.659919 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.659918 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2wvh\" (UniqueName: \"kubernetes.io/projected/84575897-3456-4e6a-9469-312424133486-kube-api-access-s2wvh\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:38:50.660094 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:50.659928 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84575897-3456-4e6a-9469-312424133486-tls-certs\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:38:51.073031 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:51.072994 2579 generic.go:356] "Generic (PLEG): container finished" podID="84575897-3456-4e6a-9469-312424133486" containerID="729fd032d9c197ee84539cf20c70157a4729b0c013213a1b32b270ddc495076e" exitCode=0 Jul 30 15:38:51.073202 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:51.073036 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" Jul 30 15:38:51.073202 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:51.073076 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" event={"ID":"84575897-3456-4e6a-9469-312424133486","Type":"ContainerDied","Data":"729fd032d9c197ee84539cf20c70157a4729b0c013213a1b32b270ddc495076e"} Jul 30 15:38:51.073202 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:51.073120 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n" event={"ID":"84575897-3456-4e6a-9469-312424133486","Type":"ContainerDied","Data":"d8f49331e7db436b2aa2ad21aaa7511ae69a3f02fb971a1462bd65453b70a75e"} Jul 30 15:38:51.073202 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:51.073138 2579 scope.go:117] "RemoveContainer" containerID="729fd032d9c197ee84539cf20c70157a4729b0c013213a1b32b270ddc495076e" Jul 30 15:38:51.083256 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:51.083240 2579 scope.go:117] "RemoveContainer" containerID="729fd032d9c197ee84539cf20c70157a4729b0c013213a1b32b270ddc495076e" Jul 30 15:38:51.083487 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:38:51.083471 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729fd032d9c197ee84539cf20c70157a4729b0c013213a1b32b270ddc495076e\": container with ID starting with 729fd032d9c197ee84539cf20c70157a4729b0c013213a1b32b270ddc495076e not found: ID does not exist" containerID="729fd032d9c197ee84539cf20c70157a4729b0c013213a1b32b270ddc495076e" Jul 30 15:38:51.083552 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:51.083495 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729fd032d9c197ee84539cf20c70157a4729b0c013213a1b32b270ddc495076e"} err="failed to get container status \"729fd032d9c197ee84539cf20c70157a4729b0c013213a1b32b270ddc495076e\": rpc error: code = NotFound desc = could not find container \"729fd032d9c197ee84539cf20c70157a4729b0c013213a1b32b270ddc495076e\": container with ID starting with 729fd032d9c197ee84539cf20c70157a4729b0c013213a1b32b270ddc495076e not found: ID does not exist" Jul 30 15:38:51.094838 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:51.094815 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n"] Jul 30 15:38:51.099146 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:51.099127 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-515950da-kserve-56db8f65f9-59b9n"] Jul 30 15:38:51.197310 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:51.197279 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84575897-3456-4e6a-9469-312424133486" path="/var/lib/kubelet/pods/84575897-3456-4e6a-9469-312424133486/volumes" Jul 30 15:38:51.866581 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:51.866520 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" podUID="81d39032-e166-46a5-afb7-fd2dd0bf4145" containerName="main" containerID="cri-o://48e4f73d1d4da36a0025e20a93596f354e298fa030dfa88a0a3b9b6475a20540" gracePeriod=45 Jul 30 15:38:52.079301 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.079267 2579 generic.go:356] "Generic (PLEG): container finished" podID="81d39032-e166-46a5-afb7-fd2dd0bf4145" containerID="48e4f73d1d4da36a0025e20a93596f354e298fa030dfa88a0a3b9b6475a20540" exitCode=0 Jul 30 15:38:52.079475 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.079346 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" event={"ID":"81d39032-e166-46a5-afb7-fd2dd0bf4145","Type":"ContainerDied","Data":"48e4f73d1d4da36a0025e20a93596f354e298fa030dfa88a0a3b9b6475a20540"} Jul 30 15:38:52.120510 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.120457 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:52.176361 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.176331 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-home\") pod \"81d39032-e166-46a5-afb7-fd2dd0bf4145\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " Jul 30 15:38:52.176542 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.176396 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-model-cache\") pod \"81d39032-e166-46a5-afb7-fd2dd0bf4145\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " Jul 30 15:38:52.176542 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.176422 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-tmp-dir\") pod \"81d39032-e166-46a5-afb7-fd2dd0bf4145\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " Jul 30 15:38:52.176542 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.176463 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d39032-e166-46a5-afb7-fd2dd0bf4145-tls-certs\") pod \"81d39032-e166-46a5-afb7-fd2dd0bf4145\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " Jul 30 15:38:52.176542 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.176526 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-dshm\") pod \"81d39032-e166-46a5-afb7-fd2dd0bf4145\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " Jul 30 15:38:52.176773 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.176578 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtdv9\" (UniqueName: \"kubernetes.io/projected/81d39032-e166-46a5-afb7-fd2dd0bf4145-kube-api-access-mtdv9\") pod \"81d39032-e166-46a5-afb7-fd2dd0bf4145\" (UID: \"81d39032-e166-46a5-afb7-fd2dd0bf4145\") " Jul 30 15:38:52.176773 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.176628 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-home" (OuterVolumeSpecName: "home") pod "81d39032-e166-46a5-afb7-fd2dd0bf4145" (UID: "81d39032-e166-46a5-afb7-fd2dd0bf4145"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:38:52.176773 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.176693 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81d39032-e166-46a5-afb7-fd2dd0bf4145" (UID: "81d39032-e166-46a5-afb7-fd2dd0bf4145"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:38:52.176773 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.176715 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-model-cache" (OuterVolumeSpecName: "model-cache") pod "81d39032-e166-46a5-afb7-fd2dd0bf4145" (UID: "81d39032-e166-46a5-afb7-fd2dd0bf4145"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:38:52.177034 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.177016 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-home\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:38:52.177086 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.177039 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-model-cache\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:38:52.177086 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.177054 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-tmp-dir\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:38:52.178587 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.178567 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81d39032-e166-46a5-afb7-fd2dd0bf4145-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81d39032-e166-46a5-afb7-fd2dd0bf4145" (UID: "81d39032-e166-46a5-afb7-fd2dd0bf4145"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:38:52.178663 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.178611 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-dshm" (OuterVolumeSpecName: "dshm") pod "81d39032-e166-46a5-afb7-fd2dd0bf4145" (UID: "81d39032-e166-46a5-afb7-fd2dd0bf4145"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:38:52.178702 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.178680 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d39032-e166-46a5-afb7-fd2dd0bf4145-kube-api-access-mtdv9" (OuterVolumeSpecName: "kube-api-access-mtdv9") pod "81d39032-e166-46a5-afb7-fd2dd0bf4145" (UID: "81d39032-e166-46a5-afb7-fd2dd0bf4145"). InnerVolumeSpecName "kube-api-access-mtdv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:38:52.277777 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.277747 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtdv9\" (UniqueName: \"kubernetes.io/projected/81d39032-e166-46a5-afb7-fd2dd0bf4145-kube-api-access-mtdv9\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:38:52.277777 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.277775 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d39032-e166-46a5-afb7-fd2dd0bf4145-tls-certs\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:38:52.277777 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:52.277784 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d39032-e166-46a5-afb7-fd2dd0bf4145-dshm\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:38:53.089961 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:53.089933 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" Jul 30 15:38:53.090434 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:53.089930 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm" event={"ID":"81d39032-e166-46a5-afb7-fd2dd0bf4145","Type":"ContainerDied","Data":"93dcd39b785af615944d8ce627aed389dfe4d7cf9353626d2cbfb0b168f4add8"} Jul 30 15:38:53.090434 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:53.090057 2579 scope.go:117] "RemoveContainer" containerID="48e4f73d1d4da36a0025e20a93596f354e298fa030dfa88a0a3b9b6475a20540" Jul 30 15:38:53.116476 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:53.116450 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm"] Jul 30 15:38:53.119260 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:53.119237 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-515950da-kserve-6dbd695799-vnccm"] Jul 30 15:38:53.196874 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:53.196843 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81d39032-e166-46a5-afb7-fd2dd0bf4145" path="/var/lib/kubelet/pods/81d39032-e166-46a5-afb7-fd2dd0bf4145/volumes" Jul 30 15:38:53.551737 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:38:53.551701 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:39:00.536953 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:39:00.536919 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:39:12.542685 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:39:12.542654 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:39:26.843776 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:39:26.843702 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:39:41.444362 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:39:41.444326 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:39:52.945959 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:39:52.945921 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:40:01.137329 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:40:01.137294 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:40:12.440911 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:40:12.440877 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:40:21.152775 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:40:21.152739 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:40:34.858646 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:40:34.855656 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:40:44.448992 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:40:44.448959 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:40:54.350672 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:40:54.350637 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:41:03.940674 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:41:03.940594 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:41:17.645377 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:41:17.645337 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:41:26.468500 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:41:26.468467 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:41:48.744117 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:41:48.744080 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:42:01.052806 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:42:01.052748 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:42:17.150370 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:42:17.150322 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:42:32.043030 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:42:32.042949 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:42:39.948031 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:42:39.947988 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:42:55.150254 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:42:55.150216 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:43:09.853965 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:09.853930 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:43:20.045351 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:20.045320 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:43:29.853397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:29.853362 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:43:30.919412 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:30.919378 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb"] Jul 30 15:43:30.919785 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:30.919772 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81d39032-e166-46a5-afb7-fd2dd0bf4145" containerName="main" Jul 30 15:43:30.919860 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:30.919787 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="81d39032-e166-46a5-afb7-fd2dd0bf4145" containerName="main" Jul 30 15:43:30.919860 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:30.919820 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84575897-3456-4e6a-9469-312424133486" containerName="main" Jul 30 15:43:30.919860 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:30.919826 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="84575897-3456-4e6a-9469-312424133486" containerName="main" Jul 30 15:43:30.919955 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:30.919901 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="84575897-3456-4e6a-9469-312424133486" containerName="main" Jul 30 15:43:30.919955 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:30.919912 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="81d39032-e166-46a5-afb7-fd2dd0bf4145" containerName="main" Jul 30 15:43:30.923236 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:30.923216 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:30.926888 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:30.926870 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-791f7d53-openshift-default-dockercfg-9c82z\"" Jul 30 15:43:30.957876 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:30.957856 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb"] Jul 30 15:43:31.025613 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.025578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-credential-socket\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.025790 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.025640 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-workload-certs\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.025790 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.025672 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-token\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.025790 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.025711 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-workload-socket\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.025790 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.025776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-envoy\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.025988 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.025838 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-podinfo\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.025988 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.025877 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-data\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.025988 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.025911 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945d3804-285e-4c6d-9d65-6a76d3ceae92-istiod-ca-cert\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.025988 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.025939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5mzj\" (UniqueName: \"kubernetes.io/projected/945d3804-285e-4c6d-9d65-6a76d3ceae92-kube-api-access-f5mzj\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.127288 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.127260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-token\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.127440 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.127305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-workload-socket\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.127440 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.127325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-envoy\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.127440 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.127348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-podinfo\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.127440 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.127387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-data\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.127440 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.127414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945d3804-285e-4c6d-9d65-6a76d3ceae92-istiod-ca-cert\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.127748 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.127447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5mzj\" (UniqueName: \"kubernetes.io/projected/945d3804-285e-4c6d-9d65-6a76d3ceae92-kube-api-access-f5mzj\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.127748 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.127488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-credential-socket\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.127748 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.127541 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-workload-certs\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.127748 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.127718 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-workload-socket\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.128009 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.127987 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-data\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.128052 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.128036 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-credential-socket\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.128099 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.128066 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-workload-certs\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.128246 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.128226 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945d3804-285e-4c6d-9d65-6a76d3ceae92-istiod-ca-cert\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.129748 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.129726 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-envoy\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.129983 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.129966 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-podinfo\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.138592 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.138564 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-token\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.138961 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.138940 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5mzj\" (UniqueName: \"kubernetes.io/projected/945d3804-285e-4c6d-9d65-6a76d3ceae92-kube-api-access-f5mzj\") pod \"e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.233522 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.233468 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:31.370110 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.370085 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb"] Jul 30 15:43:31.371366 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:43:31.371336 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945d3804_285e_4c6d_9d65_6a76d3ceae92.slice/crio-4872f7b39e49163da56c26621164e489d36c42a9447796e33ccd08cee2233b4d WatchSource:0}: Error finding container 4872f7b39e49163da56c26621164e489d36c42a9447796e33ccd08cee2233b4d: Status 404 returned error can't find the container with id 4872f7b39e49163da56c26621164e489d36c42a9447796e33ccd08cee2233b4d Jul 30 15:43:31.373556 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.373515 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:43:31.373641 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.373614 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:43:31.373679 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.373661 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:43:31.403221 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:31.403192 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" event={"ID":"945d3804-285e-4c6d-9d65-6a76d3ceae92","Type":"ContainerStarted","Data":"4872f7b39e49163da56c26621164e489d36c42a9447796e33ccd08cee2233b4d"} Jul 30 15:43:32.410503 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:32.410475 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" event={"ID":"945d3804-285e-4c6d-9d65-6a76d3ceae92","Type":"ContainerStarted","Data":"3561ecc494fdb76a8f9fca7ecd644e60fa7142d7749547fd34b76f05adc5c3b4"} Jul 30 15:43:32.440806 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:32.440758 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" podStartSLOduration=2.4407408 podStartE2EDuration="2.4407408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:43:32.438311076 +0000 UTC m=+2039.673441921" watchObservedRunningTime="2026-07-30 15:43:32.4407408 +0000 UTC m=+2039.675871647" Jul 30 15:43:33.233958 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:33.233927 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:33.238080 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:33.238058 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:33.415055 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:33.415030 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:33.415909 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:33.415891 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:35.727270 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:35.727239 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7"] Jul 30 15:43:35.730884 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:35.730865 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7" Jul 30 15:43:35.734823 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:35.734780 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-791f7d53\"" Jul 30 15:43:35.735667 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:35.735651 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-791f7d53-dockercfg-ltxt8\"" Jul 30 15:43:35.742362 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:35.742340 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7"] Jul 30 15:43:35.872560 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:35.872524 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwvgp\" (UniqueName: \"kubernetes.io/projected/8a617140-cd47-4a05-8ee6-aa5414b9c4f3-kube-api-access-nwvgp\") pod \"payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7\" (UID: \"8a617140-cd47-4a05-8ee6-aa5414b9c4f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7" Jul 30 15:43:35.872738 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:35.872605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a617140-cd47-4a05-8ee6-aa5414b9c4f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7\" (UID: \"8a617140-cd47-4a05-8ee6-aa5414b9c4f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7" Jul 30 15:43:35.973549 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:35.973517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a617140-cd47-4a05-8ee6-aa5414b9c4f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7\" (UID: \"8a617140-cd47-4a05-8ee6-aa5414b9c4f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7" Jul 30 15:43:35.973707 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:35.973685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwvgp\" (UniqueName: \"kubernetes.io/projected/8a617140-cd47-4a05-8ee6-aa5414b9c4f3-kube-api-access-nwvgp\") pod \"payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7\" (UID: \"8a617140-cd47-4a05-8ee6-aa5414b9c4f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7" Jul 30 15:43:35.974289 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:35.974264 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a617140-cd47-4a05-8ee6-aa5414b9c4f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7\" (UID: \"8a617140-cd47-4a05-8ee6-aa5414b9c4f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7" Jul 30 15:43:35.983094 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:35.983032 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwvgp\" (UniqueName: \"kubernetes.io/projected/8a617140-cd47-4a05-8ee6-aa5414b9c4f3-kube-api-access-nwvgp\") pod \"payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7\" (UID: \"8a617140-cd47-4a05-8ee6-aa5414b9c4f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7" Jul 30 15:43:36.042042 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:36.042001 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7" Jul 30 15:43:36.179728 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:36.179702 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7"] Jul 30 15:43:36.181273 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:43:36.181248 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a617140_cd47_4a05_8ee6_aa5414b9c4f3.slice/crio-bfc41fe1ec881bf884ed23d6040edbe37468dbc7bd6a78306118b3d7913e248d WatchSource:0}: Error finding container bfc41fe1ec881bf884ed23d6040edbe37468dbc7bd6a78306118b3d7913e248d: Status 404 returned error can't find the container with id bfc41fe1ec881bf884ed23d6040edbe37468dbc7bd6a78306118b3d7913e248d Jul 30 15:43:36.183245 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:36.183230 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 15:43:36.429887 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:36.429850 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7" event={"ID":"8a617140-cd47-4a05-8ee6-aa5414b9c4f3","Type":"ContainerStarted","Data":"bfc41fe1ec881bf884ed23d6040edbe37468dbc7bd6a78306118b3d7913e248d"} Jul 30 15:43:38.443494 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:38.443460 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7" event={"ID":"8a617140-cd47-4a05-8ee6-aa5414b9c4f3","Type":"ContainerStarted","Data":"3e68c9ff1322d00e319f2b87e4995ea663835f1893eb5efea17d60d9952e0603"} Jul 30 15:43:38.443886 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:38.443567 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7" Jul 30 15:43:41.243192 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:41.243094 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7" podStartSLOduration=4.225491974 podStartE2EDuration="6.243077684s" totalImagesPullingTime="2.01758571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:43:38.464101654 +0000 UTC m=+2045.699232500" watchObservedRunningTime="2026-07-30 15:43:41.243077684 +0000 UTC m=+2048.478208529" Jul 30 15:43:41.243903 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:41.243885 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:43:48.971929 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:48.971467 2579 patch_prober.go:28] interesting pod/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": read tcp 10.132.0.2:56096->10.132.0.35:15021: read: connection reset by peer" start-of-body= Jul 30 15:43:48.971929 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:48.971553 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" podUID="945d3804-285e-4c6d-9d65-6a76d3ceae92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": read tcp 10.132.0.2:56096->10.132.0.35:15021: read: connection reset by peer" Jul 30 15:43:49.450787 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:49.450754 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7" Jul 30 15:43:49.510964 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:49.510934 2579 generic.go:356] "Generic (PLEG): container finished" podID="945d3804-285e-4c6d-9d65-6a76d3ceae92" containerID="3561ecc494fdb76a8f9fca7ecd644e60fa7142d7749547fd34b76f05adc5c3b4" exitCode=0 Jul 30 15:43:49.511142 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:49.510978 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" event={"ID":"945d3804-285e-4c6d-9d65-6a76d3ceae92","Type":"ContainerDied","Data":"3561ecc494fdb76a8f9fca7ecd644e60fa7142d7749547fd34b76f05adc5c3b4"} Jul 30 15:43:49.511371 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:49.511355 2579 scope.go:117] "RemoveContainer" containerID="3561ecc494fdb76a8f9fca7ecd644e60fa7142d7749547fd34b76f05adc5c3b4" Jul 30 15:43:49.512448 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:49.512418 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:43:49.512563 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:49.512496 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:43:49.512563 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:49.512543 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:43:50.517533 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:50.517497 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" event={"ID":"945d3804-285e-4c6d-9d65-6a76d3ceae92","Type":"ContainerStarted","Data":"313ffd2f4a5871ab4ce8327a115ac28b516d64c61450492f5d81962454f4276a"} Jul 30 15:43:51.234337 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:51.234303 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:51.234533 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:51.234511 2579 patch_prober.go:28] interesting pod/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 30 15:43:51.234586 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:51.234566 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" podUID="945d3804-285e-4c6d-9d65-6a76d3ceae92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 30 15:43:52.233887 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:52.233849 2579 patch_prober.go:28] interesting pod/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 30 15:43:52.234350 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:52.233911 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" podUID="945d3804-285e-4c6d-9d65-6a76d3ceae92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 30 15:43:53.234529 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:53.234495 2579 patch_prober.go:28] interesting pod/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 30 15:43:53.234938 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:53.234549 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" podUID="945d3804-285e-4c6d-9d65-6a76d3ceae92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 30 15:43:54.234497 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:54.234460 2579 patch_prober.go:28] interesting pod/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 30 15:43:54.234670 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:54.234536 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" podUID="945d3804-285e-4c6d-9d65-6a76d3ceae92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 30 15:43:55.237368 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:55.237331 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:55.237735 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:55.237712 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:43:55.238508 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:43:55.238490 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:44:04.682642 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:04.682606 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-791f7d53-7tz52"] Jul 30 15:44:04.686387 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:04.686372 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-791f7d53-7tz52" Jul 30 15:44:04.691741 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:04.691717 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 15:44:04.691892 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:04.691780 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dg2n2\"" Jul 30 15:44:04.700640 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:04.700621 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-791f7d53-7tz52"] Jul 30 15:44:04.740313 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:04.740283 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65c942e0-3fe4-4fe3-a79e-182e2ed26139-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-791f7d53-7tz52\" (UID: \"65c942e0-3fe4-4fe3-a79e-182e2ed26139\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-791f7d53-7tz52" Jul 30 15:44:04.841779 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:04.841750 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65c942e0-3fe4-4fe3-a79e-182e2ed26139-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-791f7d53-7tz52\" (UID: \"65c942e0-3fe4-4fe3-a79e-182e2ed26139\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-791f7d53-7tz52" Jul 30 15:44:04.842349 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:04.842327 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65c942e0-3fe4-4fe3-a79e-182e2ed26139-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-791f7d53-7tz52\" (UID: \"65c942e0-3fe4-4fe3-a79e-182e2ed26139\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-791f7d53-7tz52" Jul 30 15:44:04.996126 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:04.996043 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-791f7d53-7tz52" Jul 30 15:44:05.339008 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:05.338978 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-791f7d53-7tz52"] Jul 30 15:44:05.339928 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:44:05.339902 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c942e0_3fe4_4fe3_a79e_182e2ed26139.slice/crio-11cfeaac484712be2a1b6db6888d51e388945b5a0686ebd22bb8bd9319c8b764 WatchSource:0}: Error finding container 11cfeaac484712be2a1b6db6888d51e388945b5a0686ebd22bb8bd9319c8b764: Status 404 returned error can't find the container with id 11cfeaac484712be2a1b6db6888d51e388945b5a0686ebd22bb8bd9319c8b764 Jul 30 15:44:05.587028 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:05.586950 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-791f7d53-7tz52" event={"ID":"65c942e0-3fe4-4fe3-a79e-182e2ed26139","Type":"ContainerStarted","Data":"11cfeaac484712be2a1b6db6888d51e388945b5a0686ebd22bb8bd9319c8b764"} Jul 30 15:44:11.618677 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:11.618641 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-791f7d53-7tz52" event={"ID":"65c942e0-3fe4-4fe3-a79e-182e2ed26139","Type":"ContainerStarted","Data":"5be9d2bd90ccd07888acce30bfa7d319c7072887b4950d779ead02dafee2866e"} Jul 30 15:44:11.639163 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:11.639116 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-791f7d53-7tz52" podStartSLOduration=1.916508401 podStartE2EDuration="7.639103369s" totalImagesPullingTime="5.722594968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:44:11.63899452 +0000 UTC m=+2078.874125367" watchObservedRunningTime="2026-07-30 15:44:11.639103369 +0000 UTC m=+2078.874234215" Jul 30 15:44:12.624343 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:12.624308 2579 generic.go:356] "Generic (PLEG): container finished" podID="65c942e0-3fe4-4fe3-a79e-182e2ed26139" containerID="5be9d2bd90ccd07888acce30bfa7d319c7072887b4950d779ead02dafee2866e" exitCode=0 Jul 30 15:44:12.624760 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:12.624373 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-791f7d53-7tz52" event={"ID":"65c942e0-3fe4-4fe3-a79e-182e2ed26139","Type":"ContainerDied","Data":"5be9d2bd90ccd07888acce30bfa7d319c7072887b4950d779ead02dafee2866e"} Jul 30 15:44:13.763945 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:13.763922 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-791f7d53-7tz52" Jul 30 15:44:13.924468 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:13.924391 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65c942e0-3fe4-4fe3-a79e-182e2ed26139-maas-api-service-ca\") pod \"65c942e0-3fe4-4fe3-a79e-182e2ed26139\" (UID: \"65c942e0-3fe4-4fe3-a79e-182e2ed26139\") " Jul 30 15:44:13.924728 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:13.924706 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c942e0-3fe4-4fe3-a79e-182e2ed26139-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "65c942e0-3fe4-4fe3-a79e-182e2ed26139" (UID: "65c942e0-3fe4-4fe3-a79e-182e2ed26139"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:44:14.025200 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:14.025173 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65c942e0-3fe4-4fe3-a79e-182e2ed26139-maas-api-service-ca\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:14.635466 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:14.635429 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-791f7d53-7tz52" event={"ID":"65c942e0-3fe4-4fe3-a79e-182e2ed26139","Type":"ContainerDied","Data":"11cfeaac484712be2a1b6db6888d51e388945b5a0686ebd22bb8bd9319c8b764"} Jul 30 15:44:14.635632 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:14.635476 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11cfeaac484712be2a1b6db6888d51e388945b5a0686ebd22bb8bd9319c8b764" Jul 30 15:44:14.635632 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:14.635438 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-791f7d53-7tz52" Jul 30 15:44:15.723126 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:15.723096 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7"] Jul 30 15:44:15.723539 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:15.723314 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7" podUID="8a617140-cd47-4a05-8ee6-aa5414b9c4f3" containerName="payload-pre-processing" containerID="cri-o://3e68c9ff1322d00e319f2b87e4995ea663835f1893eb5efea17d60d9952e0603" gracePeriod=30 Jul 30 15:44:15.974567 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:15.974468 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-791f7d53-7tz52"] Jul 30 15:44:15.976615 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:15.976587 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-791f7d53-7tz52"] Jul 30 15:44:15.980743 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:15.980723 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7" Jul 30 15:44:16.119503 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:16.119472 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb"] Jul 30 15:44:16.119724 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:16.119699 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" podUID="945d3804-285e-4c6d-9d65-6a76d3ceae92" containerName="istio-proxy" containerID="cri-o://313ffd2f4a5871ab4ce8327a115ac28b516d64c61450492f5d81962454f4276a" gracePeriod=30 Jul 30 15:44:16.143304 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:16.143281 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwvgp\" (UniqueName: \"kubernetes.io/projected/8a617140-cd47-4a05-8ee6-aa5414b9c4f3-kube-api-access-nwvgp\") pod \"8a617140-cd47-4a05-8ee6-aa5414b9c4f3\" (UID: \"8a617140-cd47-4a05-8ee6-aa5414b9c4f3\") " Jul 30 15:44:16.143413 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:16.143378 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a617140-cd47-4a05-8ee6-aa5414b9c4f3-plugins-config-volume\") pod \"8a617140-cd47-4a05-8ee6-aa5414b9c4f3\" (UID: \"8a617140-cd47-4a05-8ee6-aa5414b9c4f3\") " Jul 30 15:44:16.143754 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:16.143726 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a617140-cd47-4a05-8ee6-aa5414b9c4f3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8a617140-cd47-4a05-8ee6-aa5414b9c4f3" (UID: "8a617140-cd47-4a05-8ee6-aa5414b9c4f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:44:16.145666 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:16.145629 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a617140-cd47-4a05-8ee6-aa5414b9c4f3-kube-api-access-nwvgp" (OuterVolumeSpecName: "kube-api-access-nwvgp") pod "8a617140-cd47-4a05-8ee6-aa5414b9c4f3" (UID: "8a617140-cd47-4a05-8ee6-aa5414b9c4f3"). InnerVolumeSpecName "kube-api-access-nwvgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:44:16.244868 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:16.244775 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a617140-cd47-4a05-8ee6-aa5414b9c4f3-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:16.244868 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:16.244827 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwvgp\" (UniqueName: \"kubernetes.io/projected/8a617140-cd47-4a05-8ee6-aa5414b9c4f3-kube-api-access-nwvgp\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:16.649630 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:16.649593 2579 generic.go:356] "Generic (PLEG): container finished" podID="8a617140-cd47-4a05-8ee6-aa5414b9c4f3" containerID="3e68c9ff1322d00e319f2b87e4995ea663835f1893eb5efea17d60d9952e0603" exitCode=0 Jul 30 15:44:16.649822 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:16.649640 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7" Jul 30 15:44:16.649822 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:16.649675 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7" event={"ID":"8a617140-cd47-4a05-8ee6-aa5414b9c4f3","Type":"ContainerDied","Data":"3e68c9ff1322d00e319f2b87e4995ea663835f1893eb5efea17d60d9952e0603"} Jul 30 15:44:16.649822 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:16.649733 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7" event={"ID":"8a617140-cd47-4a05-8ee6-aa5414b9c4f3","Type":"ContainerDied","Data":"bfc41fe1ec881bf884ed23d6040edbe37468dbc7bd6a78306118b3d7913e248d"} Jul 30 15:44:16.649822 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:16.649760 2579 scope.go:117] "RemoveContainer" containerID="3e68c9ff1322d00e319f2b87e4995ea663835f1893eb5efea17d60d9952e0603" Jul 30 15:44:16.658554 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:16.658537 2579 scope.go:117] "RemoveContainer" containerID="3e68c9ff1322d00e319f2b87e4995ea663835f1893eb5efea17d60d9952e0603" Jul 30 15:44:16.658855 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:44:16.658833 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e68c9ff1322d00e319f2b87e4995ea663835f1893eb5efea17d60d9952e0603\": container with ID starting with 3e68c9ff1322d00e319f2b87e4995ea663835f1893eb5efea17d60d9952e0603 not found: ID does not exist" containerID="3e68c9ff1322d00e319f2b87e4995ea663835f1893eb5efea17d60d9952e0603" Jul 30 15:44:16.658946 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:16.658868 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e68c9ff1322d00e319f2b87e4995ea663835f1893eb5efea17d60d9952e0603"} err="failed to get container status \"3e68c9ff1322d00e319f2b87e4995ea663835f1893eb5efea17d60d9952e0603\": rpc error: code = NotFound desc = could not find container \"3e68c9ff1322d00e319f2b87e4995ea663835f1893eb5efea17d60d9952e0603\": container with ID starting with 3e68c9ff1322d00e319f2b87e4995ea663835f1893eb5efea17d60d9952e0603 not found: ID does not exist" Jul 30 15:44:16.679537 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:16.679514 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7"] Jul 30 15:44:16.685343 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:16.685323 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-791f7d53-84cddd8c84-7hht7"] Jul 30 15:44:17.199381 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.199340 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65c942e0-3fe4-4fe3-a79e-182e2ed26139" path="/var/lib/kubelet/pods/65c942e0-3fe4-4fe3-a79e-182e2ed26139/volumes" Jul 30 15:44:17.199959 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.199936 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a617140-cd47-4a05-8ee6-aa5414b9c4f3" path="/var/lib/kubelet/pods/8a617140-cd47-4a05-8ee6-aa5414b9c4f3/volumes" Jul 30 15:44:17.269638 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.269602 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf"] Jul 30 15:44:17.270225 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.270207 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a617140-cd47-4a05-8ee6-aa5414b9c4f3" containerName="payload-pre-processing" Jul 30 15:44:17.270304 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.270228 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a617140-cd47-4a05-8ee6-aa5414b9c4f3" containerName="payload-pre-processing" Jul 30 15:44:17.270304 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.270268 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65c942e0-3fe4-4fe3-a79e-182e2ed26139" containerName="revoke-keys" Jul 30 15:44:17.270304 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.270278 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="65c942e0-3fe4-4fe3-a79e-182e2ed26139" containerName="revoke-keys" Jul 30 15:44:17.270452 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.270360 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a617140-cd47-4a05-8ee6-aa5414b9c4f3" containerName="payload-pre-processing" Jul 30 15:44:17.270452 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.270374 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="65c942e0-3fe4-4fe3-a79e-182e2ed26139" containerName="revoke-keys" Jul 30 15:44:17.277846 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.277828 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.281237 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.281220 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-a0bccac2-gw-openshift-default-dockercfg-hfbjj\"" Jul 30 15:44:17.291548 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.291529 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf"] Jul 30 15:44:17.356050 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.356016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-workload-certs\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.356220 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.356090 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-workload-socket\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.356220 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.356121 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istiod-ca-cert\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.356220 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.356146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-envoy\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.356220 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.356165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-podinfo\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.356220 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.356184 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9drx\" (UniqueName: \"kubernetes.io/projected/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-kube-api-access-t9drx\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.356389 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.356298 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-credential-socket\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.356389 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.356331 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-data\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.356389 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.356362 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-token\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.457160 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.457079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9drx\" (UniqueName: \"kubernetes.io/projected/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-kube-api-access-t9drx\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.457160 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.457137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-credential-socket\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.457400 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.457169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-data\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.457400 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.457197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-token\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.457400 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.457280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-workload-certs\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.457400 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.457338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-workload-socket\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.457400 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.457375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istiod-ca-cert\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.457651 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.457407 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-envoy\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.457651 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.457515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-podinfo\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.457742 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.457644 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-data\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.457742 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.457671 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-credential-socket\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.457742 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.457711 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-workload-socket\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.457925 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.457752 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-workload-certs\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.458092 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.458072 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istiod-ca-cert\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.459614 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.459595 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-envoy\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.459910 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.459894 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-podinfo\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.470106 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.470079 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9drx\" (UniqueName: \"kubernetes.io/projected/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-kube-api-access-t9drx\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.476686 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.476662 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-token\") pod \"e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.588288 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.588259 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:17.735459 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.735433 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf"] Jul 30 15:44:17.737401 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:44:17.737373 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f1dd54_fd9f_4696_91f1_a0f82b5ab94f.slice/crio-c863f77148aa36447561d159b35463c28251b859d73399f75ccd070d2beab5ea WatchSource:0}: Error finding container c863f77148aa36447561d159b35463c28251b859d73399f75ccd070d2beab5ea: Status 404 returned error can't find the container with id c863f77148aa36447561d159b35463c28251b859d73399f75ccd070d2beab5ea Jul 30 15:44:17.739450 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.739414 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:44:17.739541 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.739478 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:44:17.739541 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:17.739507 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:44:18.666257 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:18.666223 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" event={"ID":"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f","Type":"ContainerStarted","Data":"eae875cf62a488b96e36f53f6f290ac488f7d42d455b924746739f7acff674fe"} Jul 30 15:44:18.666257 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:18.666273 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" event={"ID":"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f","Type":"ContainerStarted","Data":"c863f77148aa36447561d159b35463c28251b859d73399f75ccd070d2beab5ea"} Jul 30 15:44:18.691229 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:18.691180 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" podStartSLOduration=1.6911656769999999 podStartE2EDuration="1.691165677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:44:18.690275275 +0000 UTC m=+2085.925406122" watchObservedRunningTime="2026-07-30 15:44:18.691165677 +0000 UTC m=+2085.926296523" Jul 30 15:44:19.548526 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:19.548491 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:44:19.589305 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:19.589278 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:19.593916 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:19.593895 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:19.671146 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:19.671115 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:19.672020 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:19.672001 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:21.004124 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.004091 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp"] Jul 30 15:44:21.008104 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.008083 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp" Jul 30 15:44:21.011297 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.011274 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a0bccac2-dockercfg-ct8p8\"" Jul 30 15:44:21.011414 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.011401 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a0bccac2\"" Jul 30 15:44:21.023309 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.023286 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp"] Jul 30 15:44:21.092629 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.092600 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqrt6\" (UniqueName: \"kubernetes.io/projected/3a16c505-2ed5-4da8-9175-0b9c208c2b8f-kube-api-access-rqrt6\") pod \"payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp\" (UID: \"3a16c505-2ed5-4da8-9175-0b9c208c2b8f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp" Jul 30 15:44:21.092896 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.092866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a16c505-2ed5-4da8-9175-0b9c208c2b8f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp\" (UID: \"3a16c505-2ed5-4da8-9175-0b9c208c2b8f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp" Jul 30 15:44:21.193767 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.193738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a16c505-2ed5-4da8-9175-0b9c208c2b8f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp\" (UID: \"3a16c505-2ed5-4da8-9175-0b9c208c2b8f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp" Jul 30 15:44:21.193919 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.193905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqrt6\" (UniqueName: \"kubernetes.io/projected/3a16c505-2ed5-4da8-9175-0b9c208c2b8f-kube-api-access-rqrt6\") pod \"payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp\" (UID: \"3a16c505-2ed5-4da8-9175-0b9c208c2b8f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp" Jul 30 15:44:21.194489 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.194451 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a16c505-2ed5-4da8-9175-0b9c208c2b8f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp\" (UID: \"3a16c505-2ed5-4da8-9175-0b9c208c2b8f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp" Jul 30 15:44:21.205028 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.205007 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqrt6\" (UniqueName: \"kubernetes.io/projected/3a16c505-2ed5-4da8-9175-0b9c208c2b8f-kube-api-access-rqrt6\") pod \"payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp\" (UID: \"3a16c505-2ed5-4da8-9175-0b9c208c2b8f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp" Jul 30 15:44:21.319415 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.319377 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp" Jul 30 15:44:21.382924 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.382901 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:44:21.464180 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.464154 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp"] Jul 30 15:44:21.465872 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:44:21.465844 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a16c505_2ed5_4da8_9175_0b9c208c2b8f.slice/crio-ffac7c20b28b9a083d9f007032ca778e09ae1833df73035b5bfbf10a98ad9f3d WatchSource:0}: Error finding container ffac7c20b28b9a083d9f007032ca778e09ae1833df73035b5bfbf10a98ad9f3d: Status 404 returned error can't find the container with id ffac7c20b28b9a083d9f007032ca778e09ae1833df73035b5bfbf10a98ad9f3d Jul 30 15:44:21.496615 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.496594 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-podinfo\") pod \"945d3804-285e-4c6d-9d65-6a76d3ceae92\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " Jul 30 15:44:21.496726 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.496687 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-workload-socket\") pod \"945d3804-285e-4c6d-9d65-6a76d3ceae92\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " Jul 30 15:44:21.496726 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.496707 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-credential-socket\") pod \"945d3804-285e-4c6d-9d65-6a76d3ceae92\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " Jul 30 15:44:21.496841 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.496746 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5mzj\" (UniqueName: \"kubernetes.io/projected/945d3804-285e-4c6d-9d65-6a76d3ceae92-kube-api-access-f5mzj\") pod \"945d3804-285e-4c6d-9d65-6a76d3ceae92\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " Jul 30 15:44:21.496841 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.496772 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-token\") pod \"945d3804-285e-4c6d-9d65-6a76d3ceae92\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " Jul 30 15:44:21.496841 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.496829 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-workload-certs\") pod \"945d3804-285e-4c6d-9d65-6a76d3ceae92\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " Jul 30 15:44:21.496999 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.496869 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945d3804-285e-4c6d-9d65-6a76d3ceae92-istiod-ca-cert\") pod \"945d3804-285e-4c6d-9d65-6a76d3ceae92\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " Jul 30 15:44:21.496999 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.496894 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-envoy\") pod \"945d3804-285e-4c6d-9d65-6a76d3ceae92\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " Jul 30 15:44:21.496999 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.496963 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-data\") pod \"945d3804-285e-4c6d-9d65-6a76d3ceae92\" (UID: \"945d3804-285e-4c6d-9d65-6a76d3ceae92\") " Jul 30 15:44:21.497148 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.497036 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "945d3804-285e-4c6d-9d65-6a76d3ceae92" (UID: "945d3804-285e-4c6d-9d65-6a76d3ceae92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:44:21.497148 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.497046 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "945d3804-285e-4c6d-9d65-6a76d3ceae92" (UID: "945d3804-285e-4c6d-9d65-6a76d3ceae92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:44:21.497313 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.497291 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-workload-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:21.497403 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.497316 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-credential-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:21.497459 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.497423 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "945d3804-285e-4c6d-9d65-6a76d3ceae92" (UID: "945d3804-285e-4c6d-9d65-6a76d3ceae92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:44:21.497521 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.497505 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945d3804-285e-4c6d-9d65-6a76d3ceae92-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "945d3804-285e-4c6d-9d65-6a76d3ceae92" (UID: "945d3804-285e-4c6d-9d65-6a76d3ceae92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:44:21.497922 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.497897 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-data" (OuterVolumeSpecName: "istio-data") pod "945d3804-285e-4c6d-9d65-6a76d3ceae92" (UID: "945d3804-285e-4c6d-9d65-6a76d3ceae92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:44:21.499038 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.499013 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "945d3804-285e-4c6d-9d65-6a76d3ceae92" (UID: "945d3804-285e-4c6d-9d65-6a76d3ceae92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 15:44:21.499121 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.499105 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-token" (OuterVolumeSpecName: "istio-token") pod "945d3804-285e-4c6d-9d65-6a76d3ceae92" (UID: "945d3804-285e-4c6d-9d65-6a76d3ceae92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:44:21.499249 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.499227 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945d3804-285e-4c6d-9d65-6a76d3ceae92-kube-api-access-f5mzj" (OuterVolumeSpecName: "kube-api-access-f5mzj") pod "945d3804-285e-4c6d-9d65-6a76d3ceae92" (UID: "945d3804-285e-4c6d-9d65-6a76d3ceae92"). InnerVolumeSpecName "kube-api-access-f5mzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:44:21.499355 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.499340 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "945d3804-285e-4c6d-9d65-6a76d3ceae92" (UID: "945d3804-285e-4c6d-9d65-6a76d3ceae92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:44:21.598761 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.598693 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-data\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:21.598761 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.598721 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-podinfo\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:21.598761 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.598731 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5mzj\" (UniqueName: \"kubernetes.io/projected/945d3804-285e-4c6d-9d65-6a76d3ceae92-kube-api-access-f5mzj\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:21.598761 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.598740 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-token\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:21.598761 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.598749 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-workload-certs\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:21.598761 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.598756 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945d3804-285e-4c6d-9d65-6a76d3ceae92-istiod-ca-cert\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:21.598761 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.598763 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945d3804-285e-4c6d-9d65-6a76d3ceae92-istio-envoy\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:21.683171 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.683130 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp" event={"ID":"3a16c505-2ed5-4da8-9175-0b9c208c2b8f","Type":"ContainerStarted","Data":"e822ceefe62a644eea7349eb0b865a5e92bcdbd93d5114ddb27ef2dd0cd327a4"} Jul 30 15:44:21.683331 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.683197 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp" event={"ID":"3a16c505-2ed5-4da8-9175-0b9c208c2b8f","Type":"ContainerStarted","Data":"ffac7c20b28b9a083d9f007032ca778e09ae1833df73035b5bfbf10a98ad9f3d"} Jul 30 15:44:21.683331 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.683317 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp" Jul 30 15:44:21.685024 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.684999 2579 generic.go:356] "Generic (PLEG): container finished" podID="945d3804-285e-4c6d-9d65-6a76d3ceae92" containerID="313ffd2f4a5871ab4ce8327a115ac28b516d64c61450492f5d81962454f4276a" exitCode=0 Jul 30 15:44:21.685151 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.685037 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" Jul 30 15:44:21.685151 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.685072 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" event={"ID":"945d3804-285e-4c6d-9d65-6a76d3ceae92","Type":"ContainerDied","Data":"313ffd2f4a5871ab4ce8327a115ac28b516d64c61450492f5d81962454f4276a"} Jul 30 15:44:21.685151 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.685117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb" event={"ID":"945d3804-285e-4c6d-9d65-6a76d3ceae92","Type":"ContainerDied","Data":"4872f7b39e49163da56c26621164e489d36c42a9447796e33ccd08cee2233b4d"} Jul 30 15:44:21.685151 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.685139 2579 scope.go:117] "RemoveContainer" containerID="313ffd2f4a5871ab4ce8327a115ac28b516d64c61450492f5d81962454f4276a" Jul 30 15:44:21.695944 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.695918 2579 scope.go:117] "RemoveContainer" containerID="3561ecc494fdb76a8f9fca7ecd644e60fa7142d7749547fd34b76f05adc5c3b4" Jul 30 15:44:21.704740 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.704028 2579 scope.go:117] "RemoveContainer" containerID="313ffd2f4a5871ab4ce8327a115ac28b516d64c61450492f5d81962454f4276a" Jul 30 15:44:21.704740 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:44:21.704402 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313ffd2f4a5871ab4ce8327a115ac28b516d64c61450492f5d81962454f4276a\": container with ID starting with 313ffd2f4a5871ab4ce8327a115ac28b516d64c61450492f5d81962454f4276a not found: ID does not exist" containerID="313ffd2f4a5871ab4ce8327a115ac28b516d64c61450492f5d81962454f4276a" Jul 30 15:44:21.704740 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.704437 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313ffd2f4a5871ab4ce8327a115ac28b516d64c61450492f5d81962454f4276a"} err="failed to get container status \"313ffd2f4a5871ab4ce8327a115ac28b516d64c61450492f5d81962454f4276a\": rpc error: code = NotFound desc = could not find container \"313ffd2f4a5871ab4ce8327a115ac28b516d64c61450492f5d81962454f4276a\": container with ID starting with 313ffd2f4a5871ab4ce8327a115ac28b516d64c61450492f5d81962454f4276a not found: ID does not exist" Jul 30 15:44:21.704740 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.704618 2579 scope.go:117] "RemoveContainer" containerID="3561ecc494fdb76a8f9fca7ecd644e60fa7142d7749547fd34b76f05adc5c3b4" Jul 30 15:44:21.711156 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:44:21.711132 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3561ecc494fdb76a8f9fca7ecd644e60fa7142d7749547fd34b76f05adc5c3b4\": container with ID starting with 3561ecc494fdb76a8f9fca7ecd644e60fa7142d7749547fd34b76f05adc5c3b4 not found: ID does not exist" containerID="3561ecc494fdb76a8f9fca7ecd644e60fa7142d7749547fd34b76f05adc5c3b4" Jul 30 15:44:21.711245 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.711169 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3561ecc494fdb76a8f9fca7ecd644e60fa7142d7749547fd34b76f05adc5c3b4"} err="failed to get container status \"3561ecc494fdb76a8f9fca7ecd644e60fa7142d7749547fd34b76f05adc5c3b4\": rpc error: code = NotFound desc = could not find container \"3561ecc494fdb76a8f9fca7ecd644e60fa7142d7749547fd34b76f05adc5c3b4\": container with ID starting with 3561ecc494fdb76a8f9fca7ecd644e60fa7142d7749547fd34b76f05adc5c3b4 not found: ID does not exist" Jul 30 15:44:21.720946 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.720907 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp" podStartSLOduration=1.720896269 podStartE2EDuration="1.720896269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:44:21.716766468 +0000 UTC m=+2088.951897313" watchObservedRunningTime="2026-07-30 15:44:21.720896269 +0000 UTC m=+2088.956027130" Jul 30 15:44:21.744058 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.744031 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb"] Jul 30 15:44:21.755505 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:21.755478 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-791f7d53-openshift-default-75c9d849f6-jqpjb"] Jul 30 15:44:23.197910 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:23.197869 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="945d3804-285e-4c6d-9d65-6a76d3ceae92" path="/var/lib/kubelet/pods/945d3804-285e-4c6d-9d65-6a76d3ceae92/volumes" Jul 30 15:44:30.251012 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:30.250971 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:44:32.700038 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:32.700010 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp" Jul 30 15:44:33.756866 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:33.756832 2579 generic.go:356] "Generic (PLEG): container finished" podID="61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" containerID="eae875cf62a488b96e36f53f6f290ac488f7d42d455b924746739f7acff674fe" exitCode=0 Jul 30 15:44:33.757282 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:33.756896 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" event={"ID":"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f","Type":"ContainerDied","Data":"eae875cf62a488b96e36f53f6f290ac488f7d42d455b924746739f7acff674fe"} Jul 30 15:44:33.757282 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:33.757272 2579 scope.go:117] "RemoveContainer" containerID="eae875cf62a488b96e36f53f6f290ac488f7d42d455b924746739f7acff674fe" Jul 30 15:44:33.758261 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:33.758232 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:44:33.758335 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:33.758298 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:44:33.758378 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:33.758332 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:44:34.672282 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:34.672217 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:34.768649 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:34.768592 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" event={"ID":"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f","Type":"ContainerStarted","Data":"febefcb577a3c71705ea01b9e69ccc30b9a9d42e78937c1dfa9504da9e302214"} Jul 30 15:44:35.588939 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:35.588908 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:35.590047 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:35.590021 2579 patch_prober.go:28] interesting pod/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 30 15:44:35.590168 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:35.590071 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" podUID="61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 30 15:44:36.589110 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:36.589074 2579 patch_prober.go:28] interesting pod/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 30 15:44:36.589518 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:36.589146 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" podUID="61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 30 15:44:37.588731 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:37.588688 2579 patch_prober.go:28] interesting pod/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 30 15:44:37.588946 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:37.588755 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" podUID="61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 30 15:44:38.588887 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:38.588854 2579 patch_prober.go:28] interesting pod/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 30 15:44:38.589288 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:38.588910 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" podUID="61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 30 15:44:39.592991 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:39.592958 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:39.593367 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:39.593231 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:39.593823 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:39.593791 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:51.924634 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:51.924584 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp"] Jul 30 15:44:51.934650 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:51.929675 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp" podUID="3a16c505-2ed5-4da8-9175-0b9c208c2b8f" containerName="payload-processing" containerID="cri-o://e822ceefe62a644eea7349eb0b865a5e92bcdbd93d5114ddb27ef2dd0cd327a4" gracePeriod=30 Jul 30 15:44:52.183190 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:52.183130 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp" Jul 30 15:44:52.291844 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:52.291808 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqrt6\" (UniqueName: \"kubernetes.io/projected/3a16c505-2ed5-4da8-9175-0b9c208c2b8f-kube-api-access-rqrt6\") pod \"3a16c505-2ed5-4da8-9175-0b9c208c2b8f\" (UID: \"3a16c505-2ed5-4da8-9175-0b9c208c2b8f\") " Jul 30 15:44:52.292037 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:52.291903 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a16c505-2ed5-4da8-9175-0b9c208c2b8f-plugins-config-volume\") pod \"3a16c505-2ed5-4da8-9175-0b9c208c2b8f\" (UID: \"3a16c505-2ed5-4da8-9175-0b9c208c2b8f\") " Jul 30 15:44:52.292326 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:52.292294 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a16c505-2ed5-4da8-9175-0b9c208c2b8f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a16c505-2ed5-4da8-9175-0b9c208c2b8f" (UID: "3a16c505-2ed5-4da8-9175-0b9c208c2b8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:44:52.294507 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:52.294484 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a16c505-2ed5-4da8-9175-0b9c208c2b8f-kube-api-access-rqrt6" (OuterVolumeSpecName: "kube-api-access-rqrt6") pod "3a16c505-2ed5-4da8-9175-0b9c208c2b8f" (UID: "3a16c505-2ed5-4da8-9175-0b9c208c2b8f"). InnerVolumeSpecName "kube-api-access-rqrt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:44:52.328543 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:52.328517 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf"] Jul 30 15:44:52.328790 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:52.328766 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" podUID="61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" containerName="istio-proxy" containerID="cri-o://febefcb577a3c71705ea01b9e69ccc30b9a9d42e78937c1dfa9504da9e302214" gracePeriod=30 Jul 30 15:44:52.392708 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:52.392681 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a16c505-2ed5-4da8-9175-0b9c208c2b8f-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:52.392708 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:52.392704 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqrt6\" (UniqueName: \"kubernetes.io/projected/3a16c505-2ed5-4da8-9175-0b9c208c2b8f-kube-api-access-rqrt6\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:52.869143 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:52.869107 2579 generic.go:356] "Generic (PLEG): container finished" podID="3a16c505-2ed5-4da8-9175-0b9c208c2b8f" containerID="e822ceefe62a644eea7349eb0b865a5e92bcdbd93d5114ddb27ef2dd0cd327a4" exitCode=0 Jul 30 15:44:52.869337 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:52.869160 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp" Jul 30 15:44:52.869337 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:52.869199 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp" event={"ID":"3a16c505-2ed5-4da8-9175-0b9c208c2b8f","Type":"ContainerDied","Data":"e822ceefe62a644eea7349eb0b865a5e92bcdbd93d5114ddb27ef2dd0cd327a4"} Jul 30 15:44:52.869337 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:52.869256 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp" event={"ID":"3a16c505-2ed5-4da8-9175-0b9c208c2b8f","Type":"ContainerDied","Data":"ffac7c20b28b9a083d9f007032ca778e09ae1833df73035b5bfbf10a98ad9f3d"} Jul 30 15:44:52.869337 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:52.869283 2579 scope.go:117] "RemoveContainer" containerID="e822ceefe62a644eea7349eb0b865a5e92bcdbd93d5114ddb27ef2dd0cd327a4" Jul 30 15:44:52.878229 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:52.878212 2579 scope.go:117] "RemoveContainer" containerID="e822ceefe62a644eea7349eb0b865a5e92bcdbd93d5114ddb27ef2dd0cd327a4" Jul 30 15:44:52.878481 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:44:52.878462 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e822ceefe62a644eea7349eb0b865a5e92bcdbd93d5114ddb27ef2dd0cd327a4\": container with ID starting with e822ceefe62a644eea7349eb0b865a5e92bcdbd93d5114ddb27ef2dd0cd327a4 not found: ID does not exist" containerID="e822ceefe62a644eea7349eb0b865a5e92bcdbd93d5114ddb27ef2dd0cd327a4" Jul 30 15:44:52.878549 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:52.878490 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e822ceefe62a644eea7349eb0b865a5e92bcdbd93d5114ddb27ef2dd0cd327a4"} err="failed to get container status \"e822ceefe62a644eea7349eb0b865a5e92bcdbd93d5114ddb27ef2dd0cd327a4\": rpc error: code = NotFound desc = could not find container \"e822ceefe62a644eea7349eb0b865a5e92bcdbd93d5114ddb27ef2dd0cd327a4\": container with ID starting with e822ceefe62a644eea7349eb0b865a5e92bcdbd93d5114ddb27ef2dd0cd327a4 not found: ID does not exist" Jul 30 15:44:52.898757 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:52.898737 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp"] Jul 30 15:44:52.901388 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:52.901367 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0bccac2-94bbf58cc-zjlhp"] Jul 30 15:44:53.200648 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.200577 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a16c505-2ed5-4da8-9175-0b9c208c2b8f" path="/var/lib/kubelet/pods/3a16c505-2ed5-4da8-9175-0b9c208c2b8f/volumes" Jul 30 15:44:53.261252 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.257624 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq"] Jul 30 15:44:53.261252 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.258497 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945d3804-285e-4c6d-9d65-6a76d3ceae92" containerName="istio-proxy" Jul 30 15:44:53.261252 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.258525 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="945d3804-285e-4c6d-9d65-6a76d3ceae92" containerName="istio-proxy" Jul 30 15:44:53.261252 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.258545 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a16c505-2ed5-4da8-9175-0b9c208c2b8f" containerName="payload-processing" Jul 30 15:44:53.261252 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.258555 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a16c505-2ed5-4da8-9175-0b9c208c2b8f" containerName="payload-processing" Jul 30 15:44:53.261252 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.258579 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945d3804-285e-4c6d-9d65-6a76d3ceae92" containerName="istio-proxy" Jul 30 15:44:53.261252 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.258589 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="945d3804-285e-4c6d-9d65-6a76d3ceae92" containerName="istio-proxy" Jul 30 15:44:53.261252 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.258770 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a16c505-2ed5-4da8-9175-0b9c208c2b8f" containerName="payload-processing" Jul 30 15:44:53.261252 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.258840 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="945d3804-285e-4c6d-9d65-6a76d3ceae92" containerName="istio-proxy" Jul 30 15:44:53.261252 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.258854 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="945d3804-285e-4c6d-9d65-6a76d3ceae92" containerName="istio-proxy" Jul 30 15:44:53.264676 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.264652 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.268349 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.268332 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-23057140-openshift-default-dockercfg-xbpwk\"" Jul 30 15:44:53.298119 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.298091 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq"] Jul 30 15:44:53.401092 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.401064 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-data\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.401295 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.401099 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-podinfo\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.401295 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.401131 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-workload-certs\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.401295 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.401244 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-envoy\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.401491 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.401307 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-workload-socket\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.401491 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.401330 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-token\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.401491 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.401395 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c707ff-3887-43c1-9165-1fdb05de2fbb-istiod-ca-cert\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.401491 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.401447 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-credential-socket\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.401652 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.401501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjc64\" (UniqueName: \"kubernetes.io/projected/94c707ff-3887-43c1-9165-1fdb05de2fbb-kube-api-access-xjc64\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.502990 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.502898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-envoy\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.502990 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.502954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-workload-socket\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.502990 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.502980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-token\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.503253 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.503032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c707ff-3887-43c1-9165-1fdb05de2fbb-istiod-ca-cert\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.503253 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.503080 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-credential-socket\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.503253 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.503113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjc64\" (UniqueName: \"kubernetes.io/projected/94c707ff-3887-43c1-9165-1fdb05de2fbb-kube-api-access-xjc64\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.503253 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.503157 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-data\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.503253 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.503197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-podinfo\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.503253 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.503246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-workload-certs\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.503562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.503402 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-workload-socket\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.503562 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.503490 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-credential-socket\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.503677 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.503569 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-data\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.503738 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.503670 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-workload-certs\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.503738 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.503720 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c707ff-3887-43c1-9165-1fdb05de2fbb-istiod-ca-cert\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.505259 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.505234 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-envoy\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.505593 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.505574 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-podinfo\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.515191 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.515163 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-token\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.515339 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.515318 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjc64\" (UniqueName: \"kubernetes.io/projected/94c707ff-3887-43c1-9165-1fdb05de2fbb-kube-api-access-xjc64\") pod \"e2e-ait-23057140-openshift-default-784c595d96-2j9zq\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.575031 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.574999 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:53.717611 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.717584 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq"] Jul 30 15:44:53.719615 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:44:53.719588 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c707ff_3887_43c1_9165_1fdb05de2fbb.slice/crio-f33d9edb1c85c73992e47250ed208712219b718f8bd775028c932f8af026233b WatchSource:0}: Error finding container f33d9edb1c85c73992e47250ed208712219b718f8bd775028c932f8af026233b: Status 404 returned error can't find the container with id f33d9edb1c85c73992e47250ed208712219b718f8bd775028c932f8af026233b Jul 30 15:44:53.721830 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.721778 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:44:53.721958 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.721876 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:44:53.721958 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.721920 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:44:53.877724 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.877686 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" event={"ID":"94c707ff-3887-43c1-9165-1fdb05de2fbb","Type":"ContainerStarted","Data":"f1fbc3e748d329d3a5ee50055dfad569a3021a25bf35addfb087c8db610ef92e"} Jul 30 15:44:53.877940 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.877737 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" event={"ID":"94c707ff-3887-43c1-9165-1fdb05de2fbb","Type":"ContainerStarted","Data":"f33d9edb1c85c73992e47250ed208712219b718f8bd775028c932f8af026233b"} Jul 30 15:44:53.903143 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:53.903088 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" podStartSLOduration=0.903071639 podStartE2EDuration="903.071639ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:44:53.899454108 +0000 UTC m=+2121.134584956" watchObservedRunningTime="2026-07-30 15:44:53.903071639 +0000 UTC m=+2121.138202486" Jul 30 15:44:54.576210 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:54.576172 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:54.581389 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:54.581352 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:54.595056 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:54.595026 2579 patch_prober.go:28] interesting pod/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 15:44:54.595190 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:54.595079 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" podUID="61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 15:44:54.884142 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:54.884106 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:54.885068 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:54.885049 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:44:56.253546 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:56.253512 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:44:57.186061 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.186026 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr"] Jul 30 15:44:57.189881 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.189864 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr" Jul 30 15:44:57.193394 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.193373 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-23057140\"" Jul 30 15:44:57.218435 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.215053 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr"] Jul 30 15:44:57.252269 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.252238 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9"] Jul 30 15:44:57.255756 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.255737 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9" Jul 30 15:44:57.258940 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.258921 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-23057140-dockercfg-bkwz7\"" Jul 30 15:44:57.265952 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.265931 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9"] Jul 30 15:44:57.338542 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.338508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be-plugins-config-volume\") pod \"payload-processing-e2e-ait-23057140-7cd55777b-zn7kr\" (UID: \"e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be\") " pod="openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr" Jul 30 15:44:57.338734 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.338629 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde5645f-9a73-406a-99b9-cc0708531279-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9\" (UID: \"dde5645f-9a73-406a-99b9-cc0708531279\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9" Jul 30 15:44:57.338827 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.338747 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfl9p\" (UniqueName: \"kubernetes.io/projected/dde5645f-9a73-406a-99b9-cc0708531279-kube-api-access-vfl9p\") pod \"payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9\" (UID: \"dde5645f-9a73-406a-99b9-cc0708531279\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9" Jul 30 15:44:57.338898 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.338839 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr6pf\" (UniqueName: \"kubernetes.io/projected/e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be-kube-api-access-sr6pf\") pod \"payload-processing-e2e-ait-23057140-7cd55777b-zn7kr\" (UID: \"e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be\") " pod="openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr" Jul 30 15:44:57.439964 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.439907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be-plugins-config-volume\") pod \"payload-processing-e2e-ait-23057140-7cd55777b-zn7kr\" (UID: \"e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be\") " pod="openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr" Jul 30 15:44:57.439964 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.439951 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde5645f-9a73-406a-99b9-cc0708531279-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9\" (UID: \"dde5645f-9a73-406a-99b9-cc0708531279\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9" Jul 30 15:44:57.440101 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.440020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfl9p\" (UniqueName: \"kubernetes.io/projected/dde5645f-9a73-406a-99b9-cc0708531279-kube-api-access-vfl9p\") pod \"payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9\" (UID: \"dde5645f-9a73-406a-99b9-cc0708531279\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9" Jul 30 15:44:57.440194 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.440173 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr6pf\" (UniqueName: \"kubernetes.io/projected/e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be-kube-api-access-sr6pf\") pod \"payload-processing-e2e-ait-23057140-7cd55777b-zn7kr\" (UID: \"e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be\") " pod="openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr" Jul 30 15:44:57.440651 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.440633 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be-plugins-config-volume\") pod \"payload-processing-e2e-ait-23057140-7cd55777b-zn7kr\" (UID: \"e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be\") " pod="openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr" Jul 30 15:44:57.440960 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.440939 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde5645f-9a73-406a-99b9-cc0708531279-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9\" (UID: \"dde5645f-9a73-406a-99b9-cc0708531279\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9" Jul 30 15:44:57.452672 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.452649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfl9p\" (UniqueName: \"kubernetes.io/projected/dde5645f-9a73-406a-99b9-cc0708531279-kube-api-access-vfl9p\") pod \"payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9\" (UID: \"dde5645f-9a73-406a-99b9-cc0708531279\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9" Jul 30 15:44:57.453421 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.453398 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr6pf\" (UniqueName: \"kubernetes.io/projected/e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be-kube-api-access-sr6pf\") pod \"payload-processing-e2e-ait-23057140-7cd55777b-zn7kr\" (UID: \"e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be\") " pod="openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr" Jul 30 15:44:57.501128 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.501098 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr" Jul 30 15:44:57.566333 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.566307 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9" Jul 30 15:44:57.589602 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.589576 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:57.738601 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:44:57.738565 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde5645f_9a73_406a_99b9_cc0708531279.slice/crio-501e0108308e14db5a4bc536102862b9e36a224f48cbbf42084b215a1c7133c2 WatchSource:0}: Error finding container 501e0108308e14db5a4bc536102862b9e36a224f48cbbf42084b215a1c7133c2: Status 404 returned error can't find the container with id 501e0108308e14db5a4bc536102862b9e36a224f48cbbf42084b215a1c7133c2 Jul 30 15:44:57.739074 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.739052 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9"] Jul 30 15:44:57.742649 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.742626 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-workload-certs\") pod \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " Jul 30 15:44:57.742761 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.742665 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-envoy\") pod \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " Jul 30 15:44:57.742761 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.742689 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-podinfo\") pod \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " Jul 30 15:44:57.742761 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.742716 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9drx\" (UniqueName: \"kubernetes.io/projected/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-kube-api-access-t9drx\") pod \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " Jul 30 15:44:57.742991 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.742780 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-workload-socket\") pod \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " Jul 30 15:44:57.742991 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.742871 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-data\") pod \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " Jul 30 15:44:57.742991 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.742900 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-token\") pod \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " Jul 30 15:44:57.742991 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.742908 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" (UID: "61f1dd54-fd9f-4696-91f1-a0f82b5ab94f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:44:57.742991 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.742942 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-credential-socket\") pod \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " Jul 30 15:44:57.743242 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.743019 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istiod-ca-cert\") pod \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\" (UID: \"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f\") " Jul 30 15:44:57.743242 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.743106 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" (UID: "61f1dd54-fd9f-4696-91f1-a0f82b5ab94f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:44:57.743410 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.743388 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-workload-certs\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:57.743473 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.743417 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-workload-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:57.743473 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.743426 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" (UID: "61f1dd54-fd9f-4696-91f1-a0f82b5ab94f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:44:57.743685 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.743653 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" (UID: "61f1dd54-fd9f-4696-91f1-a0f82b5ab94f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:44:57.743747 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.743722 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-data" (OuterVolumeSpecName: "istio-data") pod "61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" (UID: "61f1dd54-fd9f-4696-91f1-a0f82b5ab94f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:44:57.744821 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.744787 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" (UID: "61f1dd54-fd9f-4696-91f1-a0f82b5ab94f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:44:57.745019 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.744997 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" (UID: "61f1dd54-fd9f-4696-91f1-a0f82b5ab94f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 15:44:57.745252 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.745234 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-kube-api-access-t9drx" (OuterVolumeSpecName: "kube-api-access-t9drx") pod "61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" (UID: "61f1dd54-fd9f-4696-91f1-a0f82b5ab94f"). InnerVolumeSpecName "kube-api-access-t9drx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:44:57.745302 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.745248 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-token" (OuterVolumeSpecName: "istio-token") pod "61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" (UID: "61f1dd54-fd9f-4696-91f1-a0f82b5ab94f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:44:57.844234 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.844211 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istiod-ca-cert\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:57.844234 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.844232 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-envoy\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:57.844400 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.844243 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-podinfo\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:57.844400 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.844257 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9drx\" (UniqueName: \"kubernetes.io/projected/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-kube-api-access-t9drx\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:57.844400 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.844270 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-data\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:57.844400 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.844282 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-istio-token\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:57.844400 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.844290 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f-credential-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:44:57.869543 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.869520 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr"] Jul 30 15:44:57.870195 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:44:57.870171 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e96123_c1b9_4ff6_8ae4_2f7a7c7e78be.slice/crio-04626a0c6b51b926921fa934512955f5acb3bff9dea65aa81ff840cc2c3ebdc9 WatchSource:0}: Error finding container 04626a0c6b51b926921fa934512955f5acb3bff9dea65aa81ff840cc2c3ebdc9: Status 404 returned error can't find the container with id 04626a0c6b51b926921fa934512955f5acb3bff9dea65aa81ff840cc2c3ebdc9 Jul 30 15:44:57.898848 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.898819 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr" event={"ID":"e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be","Type":"ContainerStarted","Data":"04626a0c6b51b926921fa934512955f5acb3bff9dea65aa81ff840cc2c3ebdc9"} Jul 30 15:44:57.900266 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.900244 2579 generic.go:356] "Generic (PLEG): container finished" podID="61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" containerID="febefcb577a3c71705ea01b9e69ccc30b9a9d42e78937c1dfa9504da9e302214" exitCode=0 Jul 30 15:44:57.900363 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.900297 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" Jul 30 15:44:57.900363 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.900325 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" event={"ID":"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f","Type":"ContainerDied","Data":"febefcb577a3c71705ea01b9e69ccc30b9a9d42e78937c1dfa9504da9e302214"} Jul 30 15:44:57.900477 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.900388 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf" event={"ID":"61f1dd54-fd9f-4696-91f1-a0f82b5ab94f","Type":"ContainerDied","Data":"c863f77148aa36447561d159b35463c28251b859d73399f75ccd070d2beab5ea"} Jul 30 15:44:57.900477 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.900423 2579 scope.go:117] "RemoveContainer" containerID="febefcb577a3c71705ea01b9e69ccc30b9a9d42e78937c1dfa9504da9e302214" Jul 30 15:44:57.901896 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.901875 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9" event={"ID":"dde5645f-9a73-406a-99b9-cc0708531279","Type":"ContainerStarted","Data":"a61978dd101680a7770ae2168733e66d89081a5f98460afb6b5681edee94c1e0"} Jul 30 15:44:57.901990 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.901916 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9" event={"ID":"dde5645f-9a73-406a-99b9-cc0708531279","Type":"ContainerStarted","Data":"501e0108308e14db5a4bc536102862b9e36a224f48cbbf42084b215a1c7133c2"} Jul 30 15:44:57.901990 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.901937 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9" Jul 30 15:44:57.911491 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.911475 2579 scope.go:117] "RemoveContainer" containerID="eae875cf62a488b96e36f53f6f290ac488f7d42d455b924746739f7acff674fe" Jul 30 15:44:57.922293 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.922255 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9" podStartSLOduration=0.922241755 podStartE2EDuration="922.241755ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:44:57.921171072 +0000 UTC m=+2125.156301919" watchObservedRunningTime="2026-07-30 15:44:57.922241755 +0000 UTC m=+2125.157372601" Jul 30 15:44:57.931121 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.931100 2579 scope.go:117] "RemoveContainer" containerID="febefcb577a3c71705ea01b9e69ccc30b9a9d42e78937c1dfa9504da9e302214" Jul 30 15:44:57.931389 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:44:57.931367 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"febefcb577a3c71705ea01b9e69ccc30b9a9d42e78937c1dfa9504da9e302214\": container with ID starting with febefcb577a3c71705ea01b9e69ccc30b9a9d42e78937c1dfa9504da9e302214 not found: ID does not exist" containerID="febefcb577a3c71705ea01b9e69ccc30b9a9d42e78937c1dfa9504da9e302214" Jul 30 15:44:57.931455 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.931398 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"febefcb577a3c71705ea01b9e69ccc30b9a9d42e78937c1dfa9504da9e302214"} err="failed to get container status \"febefcb577a3c71705ea01b9e69ccc30b9a9d42e78937c1dfa9504da9e302214\": rpc error: code = NotFound desc = could not find container \"febefcb577a3c71705ea01b9e69ccc30b9a9d42e78937c1dfa9504da9e302214\": container with ID starting with febefcb577a3c71705ea01b9e69ccc30b9a9d42e78937c1dfa9504da9e302214 not found: ID does not exist" Jul 30 15:44:57.931455 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.931420 2579 scope.go:117] "RemoveContainer" containerID="eae875cf62a488b96e36f53f6f290ac488f7d42d455b924746739f7acff674fe" Jul 30 15:44:57.931688 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:44:57.931668 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eae875cf62a488b96e36f53f6f290ac488f7d42d455b924746739f7acff674fe\": container with ID starting with eae875cf62a488b96e36f53f6f290ac488f7d42d455b924746739f7acff674fe not found: ID does not exist" containerID="eae875cf62a488b96e36f53f6f290ac488f7d42d455b924746739f7acff674fe" Jul 30 15:44:57.931751 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.931695 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae875cf62a488b96e36f53f6f290ac488f7d42d455b924746739f7acff674fe"} err="failed to get container status \"eae875cf62a488b96e36f53f6f290ac488f7d42d455b924746739f7acff674fe\": rpc error: code = NotFound desc = could not find container \"eae875cf62a488b96e36f53f6f290ac488f7d42d455b924746739f7acff674fe\": container with ID starting with eae875cf62a488b96e36f53f6f290ac488f7d42d455b924746739f7acff674fe not found: ID does not exist" Jul 30 15:44:57.936307 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.936287 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf"] Jul 30 15:44:57.941068 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:57.941050 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-a0bccac2-gw-openshift-default-74c94b84d4-hhknf"] Jul 30 15:44:58.908600 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:58.908564 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr" event={"ID":"e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be","Type":"ContainerStarted","Data":"d5e6239e3ca762d5ba49460a0eebbd41917339d0dd3241fdf033f3a77ef2860a"} Jul 30 15:44:58.909092 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:58.908661 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr" Jul 30 15:44:58.929084 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:58.929040 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr" podStartSLOduration=1.929025226 podStartE2EDuration="1.929025226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:44:58.928062141 +0000 UTC m=+2126.163192988" watchObservedRunningTime="2026-07-30 15:44:58.929025226 +0000 UTC m=+2126.164156049" Jul 30 15:44:59.198216 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:44:59.198133 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" path="/var/lib/kubelet/pods/61f1dd54-fd9f-4696-91f1-a0f82b5ab94f/volumes" Jul 30 15:45:06.257675 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:06.257616 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:45:08.912744 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:08.912715 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9" Jul 30 15:45:09.916487 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:09.916458 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr" Jul 30 15:45:10.086364 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:10.086224 2579 patch_prober.go:28] interesting pod/e2e-ait-23057140-openshift-default-784c595d96-2j9zq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": read tcp 10.132.0.2:58254->10.132.0.40:15021: read: connection reset by peer" start-of-body= Jul 30 15:45:10.086364 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:10.086299 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" podUID="94c707ff-3887-43c1-9165-1fdb05de2fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": read tcp 10.132.0.2:58254->10.132.0.40:15021: read: connection reset by peer" Jul 30 15:45:10.975764 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:10.975731 2579 generic.go:356] "Generic (PLEG): container finished" podID="94c707ff-3887-43c1-9165-1fdb05de2fbb" containerID="f1fbc3e748d329d3a5ee50055dfad569a3021a25bf35addfb087c8db610ef92e" exitCode=0 Jul 30 15:45:10.975764 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:10.975768 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" event={"ID":"94c707ff-3887-43c1-9165-1fdb05de2fbb","Type":"ContainerDied","Data":"f1fbc3e748d329d3a5ee50055dfad569a3021a25bf35addfb087c8db610ef92e"} Jul 30 15:45:10.976177 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:10.976136 2579 scope.go:117] "RemoveContainer" containerID="f1fbc3e748d329d3a5ee50055dfad569a3021a25bf35addfb087c8db610ef92e" Jul 30 15:45:10.977136 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:10.977105 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:45:10.977212 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:10.977194 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:45:10.977283 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:10.977237 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:45:11.983112 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:11.983079 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" event={"ID":"94c707ff-3887-43c1-9165-1fdb05de2fbb","Type":"ContainerStarted","Data":"a5082090ea7de936c3ddd2e7307395d07978cb3eacdc514858577a22096bcb75"} Jul 30 15:45:12.575368 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:12.575320 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:45:12.575583 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:12.575541 2579 patch_prober.go:28] interesting pod/e2e-ait-23057140-openshift-default-784c595d96-2j9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 30 15:45:12.575659 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:12.575585 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" podUID="94c707ff-3887-43c1-9165-1fdb05de2fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 30 15:45:13.575485 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:13.575450 2579 patch_prober.go:28] interesting pod/e2e-ait-23057140-openshift-default-784c595d96-2j9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 30 15:45:13.575892 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:13.575518 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" podUID="94c707ff-3887-43c1-9165-1fdb05de2fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 30 15:45:14.576146 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:14.576110 2579 patch_prober.go:28] interesting pod/e2e-ait-23057140-openshift-default-784c595d96-2j9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 30 15:45:14.576612 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:14.576186 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" podUID="94c707ff-3887-43c1-9165-1fdb05de2fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 30 15:45:15.575773 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:15.575739 2579 patch_prober.go:28] interesting pod/e2e-ait-23057140-openshift-default-784c595d96-2j9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 30 15:45:15.576111 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:15.576078 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" podUID="94c707ff-3887-43c1-9165-1fdb05de2fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 30 15:45:16.579138 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:16.579109 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:45:16.579604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:16.579373 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:45:16.580020 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:16.580002 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:45:27.068930 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:27.068847 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr"] Jul 30 15:45:27.069734 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:27.069244 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr" podUID="e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be" containerName="payload-processing" containerID="cri-o://d5e6239e3ca762d5ba49460a0eebbd41917339d0dd3241fdf033f3a77ef2860a" gracePeriod=30 Jul 30 15:45:27.077852 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:27.077825 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9"] Jul 30 15:45:27.078072 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:27.078030 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9" podUID="dde5645f-9a73-406a-99b9-cc0708531279" containerName="payload-pre-processing" containerID="cri-o://a61978dd101680a7770ae2168733e66d89081a5f98460afb6b5681edee94c1e0" gracePeriod=30 Jul 30 15:45:27.352659 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:27.352631 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr" Jul 30 15:45:27.355457 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:27.355439 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9" Jul 30 15:45:27.416746 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:27.416719 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde5645f-9a73-406a-99b9-cc0708531279-plugins-config-volume\") pod \"dde5645f-9a73-406a-99b9-cc0708531279\" (UID: \"dde5645f-9a73-406a-99b9-cc0708531279\") " Jul 30 15:45:27.416901 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:27.416761 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfl9p\" (UniqueName: \"kubernetes.io/projected/dde5645f-9a73-406a-99b9-cc0708531279-kube-api-access-vfl9p\") pod \"dde5645f-9a73-406a-99b9-cc0708531279\" (UID: \"dde5645f-9a73-406a-99b9-cc0708531279\") " Jul 30 15:45:27.416901 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:27.416884 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sr6pf\" (UniqueName: \"kubernetes.io/projected/e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be-kube-api-access-sr6pf\") pod \"e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be\" (UID: \"e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be\") " Jul 30 15:45:27.417021 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:27.416947 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be-plugins-config-volume\") pod \"e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be\" (UID: \"e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be\") " Jul 30 15:45:27.417100 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:27.417077 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dde5645f-9a73-406a-99b9-cc0708531279-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dde5645f-9a73-406a-99b9-cc0708531279" (UID: "dde5645f-9a73-406a-99b9-cc0708531279"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:45:27.417304 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:27.417281 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be" (UID: "e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:45:27.417385 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:27.417317 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde5645f-9a73-406a-99b9-cc0708531279-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:45:27.418859 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:27.418837 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde5645f-9a73-406a-99b9-cc0708531279-kube-api-access-vfl9p" (OuterVolumeSpecName: "kube-api-access-vfl9p") pod "dde5645f-9a73-406a-99b9-cc0708531279" (UID: "dde5645f-9a73-406a-99b9-cc0708531279"). InnerVolumeSpecName "kube-api-access-vfl9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:45:27.418962 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:27.418852 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be-kube-api-access-sr6pf" (OuterVolumeSpecName: "kube-api-access-sr6pf") pod "e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be" (UID: "e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be"). InnerVolumeSpecName "kube-api-access-sr6pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:45:27.517961 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:27.517936 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfl9p\" (UniqueName: \"kubernetes.io/projected/dde5645f-9a73-406a-99b9-cc0708531279-kube-api-access-vfl9p\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:45:27.517961 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:27.517960 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr6pf\" (UniqueName: \"kubernetes.io/projected/e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be-kube-api-access-sr6pf\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:45:27.518130 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:27.517969 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:45:28.067450 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.067414 2579 generic.go:356] "Generic (PLEG): container finished" podID="e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be" containerID="d5e6239e3ca762d5ba49460a0eebbd41917339d0dd3241fdf033f3a77ef2860a" exitCode=0 Jul 30 15:45:28.067598 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.067457 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr" Jul 30 15:45:28.067598 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.067493 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr" event={"ID":"e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be","Type":"ContainerDied","Data":"d5e6239e3ca762d5ba49460a0eebbd41917339d0dd3241fdf033f3a77ef2860a"} Jul 30 15:45:28.067598 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.067548 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr" event={"ID":"e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be","Type":"ContainerDied","Data":"04626a0c6b51b926921fa934512955f5acb3bff9dea65aa81ff840cc2c3ebdc9"} Jul 30 15:45:28.067598 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.067573 2579 scope.go:117] "RemoveContainer" containerID="d5e6239e3ca762d5ba49460a0eebbd41917339d0dd3241fdf033f3a77ef2860a" Jul 30 15:45:28.068922 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.068898 2579 generic.go:356] "Generic (PLEG): container finished" podID="dde5645f-9a73-406a-99b9-cc0708531279" containerID="a61978dd101680a7770ae2168733e66d89081a5f98460afb6b5681edee94c1e0" exitCode=0 Jul 30 15:45:28.069030 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.068933 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9" Jul 30 15:45:28.069030 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.068956 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9" event={"ID":"dde5645f-9a73-406a-99b9-cc0708531279","Type":"ContainerDied","Data":"a61978dd101680a7770ae2168733e66d89081a5f98460afb6b5681edee94c1e0"} Jul 30 15:45:28.069030 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.068997 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9" event={"ID":"dde5645f-9a73-406a-99b9-cc0708531279","Type":"ContainerDied","Data":"501e0108308e14db5a4bc536102862b9e36a224f48cbbf42084b215a1c7133c2"} Jul 30 15:45:28.079854 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.079611 2579 scope.go:117] "RemoveContainer" containerID="d5e6239e3ca762d5ba49460a0eebbd41917339d0dd3241fdf033f3a77ef2860a" Jul 30 15:45:28.079939 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:45:28.079916 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5e6239e3ca762d5ba49460a0eebbd41917339d0dd3241fdf033f3a77ef2860a\": container with ID starting with d5e6239e3ca762d5ba49460a0eebbd41917339d0dd3241fdf033f3a77ef2860a not found: ID does not exist" containerID="d5e6239e3ca762d5ba49460a0eebbd41917339d0dd3241fdf033f3a77ef2860a" Jul 30 15:45:28.080005 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.079950 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5e6239e3ca762d5ba49460a0eebbd41917339d0dd3241fdf033f3a77ef2860a"} err="failed to get container status \"d5e6239e3ca762d5ba49460a0eebbd41917339d0dd3241fdf033f3a77ef2860a\": rpc error: code = NotFound desc = could not find container \"d5e6239e3ca762d5ba49460a0eebbd41917339d0dd3241fdf033f3a77ef2860a\": container with ID starting with d5e6239e3ca762d5ba49460a0eebbd41917339d0dd3241fdf033f3a77ef2860a not found: ID does not exist" Jul 30 15:45:28.080005 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.079973 2579 scope.go:117] "RemoveContainer" containerID="a61978dd101680a7770ae2168733e66d89081a5f98460afb6b5681edee94c1e0" Jul 30 15:45:28.089003 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.088987 2579 scope.go:117] "RemoveContainer" containerID="a61978dd101680a7770ae2168733e66d89081a5f98460afb6b5681edee94c1e0" Jul 30 15:45:28.089241 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:45:28.089225 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61978dd101680a7770ae2168733e66d89081a5f98460afb6b5681edee94c1e0\": container with ID starting with a61978dd101680a7770ae2168733e66d89081a5f98460afb6b5681edee94c1e0 not found: ID does not exist" containerID="a61978dd101680a7770ae2168733e66d89081a5f98460afb6b5681edee94c1e0" Jul 30 15:45:28.089293 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.089248 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61978dd101680a7770ae2168733e66d89081a5f98460afb6b5681edee94c1e0"} err="failed to get container status \"a61978dd101680a7770ae2168733e66d89081a5f98460afb6b5681edee94c1e0\": rpc error: code = NotFound desc = could not find container \"a61978dd101680a7770ae2168733e66d89081a5f98460afb6b5681edee94c1e0\": container with ID starting with a61978dd101680a7770ae2168733e66d89081a5f98460afb6b5681edee94c1e0 not found: ID does not exist" Jul 30 15:45:28.104883 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.104856 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9"] Jul 30 15:45:28.108647 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.108625 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-23057140-cd8c7fc5b-4mlt9"] Jul 30 15:45:28.123504 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.123481 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr"] Jul 30 15:45:28.131324 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.131303 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23057140-7cd55777b-zn7kr"] Jul 30 15:45:28.513119 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.513089 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq"] Jul 30 15:45:28.513325 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:28.513301 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" podUID="94c707ff-3887-43c1-9165-1fdb05de2fbb" containerName="istio-proxy" containerID="cri-o://a5082090ea7de936c3ddd2e7307395d07978cb3eacdc514858577a22096bcb75" gracePeriod=30 Jul 30 15:45:29.197858 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.197825 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dde5645f-9a73-406a-99b9-cc0708531279" path="/var/lib/kubelet/pods/dde5645f-9a73-406a-99b9-cc0708531279/volumes" Jul 30 15:45:29.198222 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.198171 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be" path="/var/lib/kubelet/pods/e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be/volumes" Jul 30 15:45:29.397098 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.397065 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc"] Jul 30 15:45:29.397460 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.397447 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" containerName="istio-proxy" Jul 30 15:45:29.397503 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.397462 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" containerName="istio-proxy" Jul 30 15:45:29.397503 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.397474 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dde5645f-9a73-406a-99b9-cc0708531279" containerName="payload-pre-processing" Jul 30 15:45:29.397503 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.397480 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dde5645f-9a73-406a-99b9-cc0708531279" containerName="payload-pre-processing" Jul 30 15:45:29.397503 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.397499 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be" containerName="payload-processing" Jul 30 15:45:29.397634 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.397505 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be" containerName="payload-processing" Jul 30 15:45:29.397634 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.397515 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" containerName="istio-proxy" Jul 30 15:45:29.397634 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.397521 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" containerName="istio-proxy" Jul 30 15:45:29.397634 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.397572 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" containerName="istio-proxy" Jul 30 15:45:29.397634 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.397579 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dde5645f-9a73-406a-99b9-cc0708531279" containerName="payload-pre-processing" Jul 30 15:45:29.397634 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.397585 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e96123-c1b9-4ff6-8ae4-2f7a7c7e78be" containerName="payload-processing" Jul 30 15:45:29.397634 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.397592 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="61f1dd54-fd9f-4696-91f1-a0f82b5ab94f" containerName="istio-proxy" Jul 30 15:45:29.400571 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.400556 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.403493 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.403471 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-d21ea6ee-openshift-default-dockercfg-mqk6j\"" Jul 30 15:45:29.418120 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.418091 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc"] Jul 30 15:45:29.537986 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.537905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-workload-socket\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.537986 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.537938 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-credential-socket\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.537986 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.537959 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-podinfo\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.538209 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.538047 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a254ff0f-3b60-44af-864d-25a8360adf3f-istiod-ca-cert\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.538209 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.538104 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-envoy\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.538209 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.538126 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-data\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.538209 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.538168 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-workload-certs\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.538209 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.538185 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4d42\" (UniqueName: \"kubernetes.io/projected/a254ff0f-3b60-44af-864d-25a8360adf3f-kube-api-access-g4d42\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.538386 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.538233 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-token\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.639217 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.639187 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-token\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.639372 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.639251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-workload-socket\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.639372 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.639267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-credential-socket\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.639372 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.639282 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-podinfo\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.639372 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.639305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a254ff0f-3b60-44af-864d-25a8360adf3f-istiod-ca-cert\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.639372 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.639336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-envoy\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.639630 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.639468 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-data\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.639630 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.639534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-workload-certs\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.639630 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.639560 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4d42\" (UniqueName: \"kubernetes.io/projected/a254ff0f-3b60-44af-864d-25a8360adf3f-kube-api-access-g4d42\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.639792 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.639710 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-credential-socket\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.639893 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.639868 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-workload-certs\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.639955 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.639938 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-workload-socket\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.640016 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.639983 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-data\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.640300 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.640276 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a254ff0f-3b60-44af-864d-25a8360adf3f-istiod-ca-cert\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.642129 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.642105 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-envoy\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.642270 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.642251 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-podinfo\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.648977 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.648952 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-token\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.649089 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.649074 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4d42\" (UniqueName: \"kubernetes.io/projected/a254ff0f-3b60-44af-864d-25a8360adf3f-kube-api-access-g4d42\") pod \"e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.715120 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.715037 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:29.849927 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.849899 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc"] Jul 30 15:45:29.850969 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:45:29.850938 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda254ff0f_3b60_44af_864d_25a8360adf3f.slice/crio-a5560abb1418308e225cbfaa32b2a2ae457931819c4f0baec6bf4211ba795a2b WatchSource:0}: Error finding container a5560abb1418308e225cbfaa32b2a2ae457931819c4f0baec6bf4211ba795a2b: Status 404 returned error can't find the container with id a5560abb1418308e225cbfaa32b2a2ae457931819c4f0baec6bf4211ba795a2b Jul 30 15:45:29.853124 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.853090 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:45:29.853219 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.853154 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:45:29.853219 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:29.853183 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:45:30.084781 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:30.084682 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" event={"ID":"a254ff0f-3b60-44af-864d-25a8360adf3f","Type":"ContainerStarted","Data":"22af1f77be97801152383869b18d49f78dea5d3258799776c12a09ac59fdd69a"} Jul 30 15:45:30.084781 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:30.084753 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" event={"ID":"a254ff0f-3b60-44af-864d-25a8360adf3f","Type":"ContainerStarted","Data":"a5560abb1418308e225cbfaa32b2a2ae457931819c4f0baec6bf4211ba795a2b"} Jul 30 15:45:30.111498 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:30.111447 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" podStartSLOduration=1.111431124 podStartE2EDuration="1.111431124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:45:30.109262718 +0000 UTC m=+2157.344393565" watchObservedRunningTime="2026-07-30 15:45:30.111431124 +0000 UTC m=+2157.346561948" Jul 30 15:45:30.715538 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:30.715504 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:30.720447 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:30.720422 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:30.936526 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:30.936490 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:45:31.090989 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:31.090913 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:31.091873 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:31.091852 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:31.581558 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:31.581521 2579 patch_prober.go:28] interesting pod/e2e-ait-23057140-openshift-default-784c595d96-2j9zq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 30 15:45:31.581728 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:31.581573 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" podUID="94c707ff-3887-43c1-9165-1fdb05de2fbb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 15:45:33.366960 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.366926 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4"] Jul 30 15:45:33.370000 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.369980 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4" Jul 30 15:45:33.377768 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.377743 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-d21ea6ee-dockercfg-dn7q2\"" Jul 30 15:45:33.378963 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.378946 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-d21ea6ee\"" Jul 30 15:45:33.388238 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.388215 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4"] Jul 30 15:45:33.475265 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.475222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40453ca-1141-43cc-a63c-63eae56fb4eb-plugins-config-volume\") pod \"payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4\" (UID: \"b40453ca-1141-43cc-a63c-63eae56fb4eb\") " pod="openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4" Jul 30 15:45:33.475425 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.475290 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvhzd\" (UniqueName: \"kubernetes.io/projected/b40453ca-1141-43cc-a63c-63eae56fb4eb-kube-api-access-xvhzd\") pod \"payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4\" (UID: \"b40453ca-1141-43cc-a63c-63eae56fb4eb\") " pod="openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4" Jul 30 15:45:33.576609 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.576575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvhzd\" (UniqueName: \"kubernetes.io/projected/b40453ca-1141-43cc-a63c-63eae56fb4eb-kube-api-access-xvhzd\") pod \"payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4\" (UID: \"b40453ca-1141-43cc-a63c-63eae56fb4eb\") " pod="openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4" Jul 30 15:45:33.576725 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.576707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40453ca-1141-43cc-a63c-63eae56fb4eb-plugins-config-volume\") pod \"payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4\" (UID: \"b40453ca-1141-43cc-a63c-63eae56fb4eb\") " pod="openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4" Jul 30 15:45:33.577415 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.577393 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40453ca-1141-43cc-a63c-63eae56fb4eb-plugins-config-volume\") pod \"payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4\" (UID: \"b40453ca-1141-43cc-a63c-63eae56fb4eb\") " pod="openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4" Jul 30 15:45:33.590137 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.590114 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvhzd\" (UniqueName: \"kubernetes.io/projected/b40453ca-1141-43cc-a63c-63eae56fb4eb-kube-api-access-xvhzd\") pod \"payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4\" (UID: \"b40453ca-1141-43cc-a63c-63eae56fb4eb\") " pod="openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4" Jul 30 15:45:33.680128 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.680096 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4" Jul 30 15:45:33.767184 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.767158 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:45:33.823032 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.823008 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4"] Jul 30 15:45:33.825698 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:45:33.825662 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40453ca_1141_43cc_a63c_63eae56fb4eb.slice/crio-60eb0baf9e8905103cb1f125c793924169ea6776a0d1d6dc8ac2fd7630f966c6 WatchSource:0}: Error finding container 60eb0baf9e8905103cb1f125c793924169ea6776a0d1d6dc8ac2fd7630f966c6: Status 404 returned error can't find the container with id 60eb0baf9e8905103cb1f125c793924169ea6776a0d1d6dc8ac2fd7630f966c6 Jul 30 15:45:33.878962 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.878944 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-workload-certs\") pod \"94c707ff-3887-43c1-9165-1fdb05de2fbb\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " Jul 30 15:45:33.879036 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.878982 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-credential-socket\") pod \"94c707ff-3887-43c1-9165-1fdb05de2fbb\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " Jul 30 15:45:33.879082 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.879041 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c707ff-3887-43c1-9165-1fdb05de2fbb-istiod-ca-cert\") pod \"94c707ff-3887-43c1-9165-1fdb05de2fbb\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " Jul 30 15:45:33.879082 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.879066 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-podinfo\") pod \"94c707ff-3887-43c1-9165-1fdb05de2fbb\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " Jul 30 15:45:33.879174 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.879093 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-envoy\") pod \"94c707ff-3887-43c1-9165-1fdb05de2fbb\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " Jul 30 15:45:33.879174 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.879126 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-data\") pod \"94c707ff-3887-43c1-9165-1fdb05de2fbb\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " Jul 30 15:45:33.879276 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.879226 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjc64\" (UniqueName: \"kubernetes.io/projected/94c707ff-3887-43c1-9165-1fdb05de2fbb-kube-api-access-xjc64\") pod \"94c707ff-3887-43c1-9165-1fdb05de2fbb\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " Jul 30 15:45:33.879276 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.879251 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-token\") pod \"94c707ff-3887-43c1-9165-1fdb05de2fbb\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " Jul 30 15:45:33.879276 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.879225 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "94c707ff-3887-43c1-9165-1fdb05de2fbb" (UID: "94c707ff-3887-43c1-9165-1fdb05de2fbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:45:33.879423 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.879314 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-workload-socket\") pod \"94c707ff-3887-43c1-9165-1fdb05de2fbb\" (UID: \"94c707ff-3887-43c1-9165-1fdb05de2fbb\") " Jul 30 15:45:33.879480 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.879441 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c707ff-3887-43c1-9165-1fdb05de2fbb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "94c707ff-3887-43c1-9165-1fdb05de2fbb" (UID: "94c707ff-3887-43c1-9165-1fdb05de2fbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:45:33.879716 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.879686 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-workload-certs\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:45:33.879716 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.879695 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "94c707ff-3887-43c1-9165-1fdb05de2fbb" (UID: "94c707ff-3887-43c1-9165-1fdb05de2fbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:45:33.879716 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.879710 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c707ff-3887-43c1-9165-1fdb05de2fbb-istiod-ca-cert\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:45:33.879944 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.879749 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "94c707ff-3887-43c1-9165-1fdb05de2fbb" (UID: "94c707ff-3887-43c1-9165-1fdb05de2fbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:45:33.880192 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.880167 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-data" (OuterVolumeSpecName: "istio-data") pod "94c707ff-3887-43c1-9165-1fdb05de2fbb" (UID: "94c707ff-3887-43c1-9165-1fdb05de2fbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:45:33.881203 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.881180 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "94c707ff-3887-43c1-9165-1fdb05de2fbb" (UID: "94c707ff-3887-43c1-9165-1fdb05de2fbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:45:33.881380 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.881351 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-token" (OuterVolumeSpecName: "istio-token") pod "94c707ff-3887-43c1-9165-1fdb05de2fbb" (UID: "94c707ff-3887-43c1-9165-1fdb05de2fbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:45:33.881513 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.881489 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "94c707ff-3887-43c1-9165-1fdb05de2fbb" (UID: "94c707ff-3887-43c1-9165-1fdb05de2fbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 15:45:33.881686 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.881666 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c707ff-3887-43c1-9165-1fdb05de2fbb-kube-api-access-xjc64" (OuterVolumeSpecName: "kube-api-access-xjc64") pod "94c707ff-3887-43c1-9165-1fdb05de2fbb" (UID: "94c707ff-3887-43c1-9165-1fdb05de2fbb"). InnerVolumeSpecName "kube-api-access-xjc64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:45:33.980656 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.980611 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjc64\" (UniqueName: \"kubernetes.io/projected/94c707ff-3887-43c1-9165-1fdb05de2fbb-kube-api-access-xjc64\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:45:33.980656 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.980650 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-token\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:45:33.980916 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.980665 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-workload-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:45:33.980916 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.980694 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-credential-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:45:33.980916 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.980709 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-podinfo\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:45:33.980916 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.980726 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-envoy\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:45:33.980916 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:33.980739 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c707ff-3887-43c1-9165-1fdb05de2fbb-istio-data\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:45:34.107163 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:34.107130 2579 generic.go:356] "Generic (PLEG): container finished" podID="94c707ff-3887-43c1-9165-1fdb05de2fbb" containerID="a5082090ea7de936c3ddd2e7307395d07978cb3eacdc514858577a22096bcb75" exitCode=0 Jul 30 15:45:34.107341 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:34.107180 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" Jul 30 15:45:34.107341 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:34.107217 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" event={"ID":"94c707ff-3887-43c1-9165-1fdb05de2fbb","Type":"ContainerDied","Data":"a5082090ea7de936c3ddd2e7307395d07978cb3eacdc514858577a22096bcb75"} Jul 30 15:45:34.107341 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:34.107280 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq" event={"ID":"94c707ff-3887-43c1-9165-1fdb05de2fbb","Type":"ContainerDied","Data":"f33d9edb1c85c73992e47250ed208712219b718f8bd775028c932f8af026233b"} Jul 30 15:45:34.107341 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:34.107310 2579 scope.go:117] "RemoveContainer" containerID="a5082090ea7de936c3ddd2e7307395d07978cb3eacdc514858577a22096bcb75" Jul 30 15:45:34.108849 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:34.108818 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4" event={"ID":"b40453ca-1141-43cc-a63c-63eae56fb4eb","Type":"ContainerStarted","Data":"dcb9bc372ef94861e76880f095bd6c36ad651848891f4fd6e709fceceeaf8414"} Jul 30 15:45:34.108967 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:34.108862 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4" event={"ID":"b40453ca-1141-43cc-a63c-63eae56fb4eb","Type":"ContainerStarted","Data":"60eb0baf9e8905103cb1f125c793924169ea6776a0d1d6dc8ac2fd7630f966c6"} Jul 30 15:45:34.108967 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:34.108909 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4" Jul 30 15:45:34.119421 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:34.119400 2579 scope.go:117] "RemoveContainer" containerID="f1fbc3e748d329d3a5ee50055dfad569a3021a25bf35addfb087c8db610ef92e" Jul 30 15:45:34.127578 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:34.127557 2579 scope.go:117] "RemoveContainer" containerID="a5082090ea7de936c3ddd2e7307395d07978cb3eacdc514858577a22096bcb75" Jul 30 15:45:34.127851 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:45:34.127832 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5082090ea7de936c3ddd2e7307395d07978cb3eacdc514858577a22096bcb75\": container with ID starting with a5082090ea7de936c3ddd2e7307395d07978cb3eacdc514858577a22096bcb75 not found: ID does not exist" containerID="a5082090ea7de936c3ddd2e7307395d07978cb3eacdc514858577a22096bcb75" Jul 30 15:45:34.127933 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:34.127864 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5082090ea7de936c3ddd2e7307395d07978cb3eacdc514858577a22096bcb75"} err="failed to get container status \"a5082090ea7de936c3ddd2e7307395d07978cb3eacdc514858577a22096bcb75\": rpc error: code = NotFound desc = could not find container \"a5082090ea7de936c3ddd2e7307395d07978cb3eacdc514858577a22096bcb75\": container with ID starting with a5082090ea7de936c3ddd2e7307395d07978cb3eacdc514858577a22096bcb75 not found: ID does not exist" Jul 30 15:45:34.127933 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:34.127889 2579 scope.go:117] "RemoveContainer" containerID="f1fbc3e748d329d3a5ee50055dfad569a3021a25bf35addfb087c8db610ef92e" Jul 30 15:45:34.128146 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:45:34.128130 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1fbc3e748d329d3a5ee50055dfad569a3021a25bf35addfb087c8db610ef92e\": container with ID starting with f1fbc3e748d329d3a5ee50055dfad569a3021a25bf35addfb087c8db610ef92e not found: ID does not exist" containerID="f1fbc3e748d329d3a5ee50055dfad569a3021a25bf35addfb087c8db610ef92e" Jul 30 15:45:34.128198 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:34.128153 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fbc3e748d329d3a5ee50055dfad569a3021a25bf35addfb087c8db610ef92e"} err="failed to get container status \"f1fbc3e748d329d3a5ee50055dfad569a3021a25bf35addfb087c8db610ef92e\": rpc error: code = NotFound desc = could not find container \"f1fbc3e748d329d3a5ee50055dfad569a3021a25bf35addfb087c8db610ef92e\": container with ID starting with f1fbc3e748d329d3a5ee50055dfad569a3021a25bf35addfb087c8db610ef92e not found: ID does not exist" Jul 30 15:45:34.137321 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:34.137253 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4" podStartSLOduration=1.137239689 podStartE2EDuration="1.137239689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:45:34.135001882 +0000 UTC m=+2161.370132727" watchObservedRunningTime="2026-07-30 15:45:34.137239689 +0000 UTC m=+2161.372370534" Jul 30 15:45:34.151676 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:34.151651 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq"] Jul 30 15:45:34.160133 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:34.160111 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-23057140-openshift-default-784c595d96-2j9zq"] Jul 30 15:45:35.197978 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:35.197942 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94c707ff-3887-43c1-9165-1fdb05de2fbb" path="/var/lib/kubelet/pods/94c707ff-3887-43c1-9165-1fdb05de2fbb/volumes" Jul 30 15:45:41.549238 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:41.549201 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:45:45.116692 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:45.116661 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4" Jul 30 15:45:46.092915 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:46.092791 2579 patch_prober.go:28] interesting pod/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 30 15:45:46.092915 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:46.092886 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" podUID="a254ff0f-3b60-44af-864d-25a8360adf3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 30 15:45:46.169538 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:46.169507 2579 generic.go:356] "Generic (PLEG): container finished" podID="a254ff0f-3b60-44af-864d-25a8360adf3f" containerID="22af1f77be97801152383869b18d49f78dea5d3258799776c12a09ac59fdd69a" exitCode=0 Jul 30 15:45:46.169877 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:46.169579 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" event={"ID":"a254ff0f-3b60-44af-864d-25a8360adf3f","Type":"ContainerDied","Data":"22af1f77be97801152383869b18d49f78dea5d3258799776c12a09ac59fdd69a"} Jul 30 15:45:46.169998 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:46.169983 2579 scope.go:117] "RemoveContainer" containerID="22af1f77be97801152383869b18d49f78dea5d3258799776c12a09ac59fdd69a" Jul 30 15:45:46.171083 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:46.171053 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:45:46.171161 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:46.171140 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:45:46.171205 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:46.171177 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:45:47.179973 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:47.179932 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" event={"ID":"a254ff0f-3b60-44af-864d-25a8360adf3f","Type":"ContainerStarted","Data":"edb15e1aef54216b791c22b20ea0b08b6ffb00264b1ed93abeb86acc61df9c64"} Jul 30 15:45:47.715996 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:47.715955 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:47.716220 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:47.716194 2579 patch_prober.go:28] interesting pod/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 30 15:45:47.716316 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:47.716270 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" podUID="a254ff0f-3b60-44af-864d-25a8360adf3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 30 15:45:48.716343 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:48.716309 2579 patch_prober.go:28] interesting pod/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 30 15:45:48.716719 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:48.716374 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" podUID="a254ff0f-3b60-44af-864d-25a8360adf3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 30 15:45:49.715678 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:49.715632 2579 patch_prober.go:28] interesting pod/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 30 15:45:49.715874 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:49.715721 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" podUID="a254ff0f-3b60-44af-864d-25a8360adf3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 30 15:45:50.716070 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:50.716032 2579 patch_prober.go:28] interesting pod/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 30 15:45:50.716487 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:50.716101 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" podUID="a254ff0f-3b60-44af-864d-25a8360adf3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 30 15:45:51.719813 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:51.719771 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:51.720193 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:51.720103 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:45:51.720656 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:45:51.720640 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:46:03.557318 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:03.557286 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4"] Jul 30 15:46:03.557847 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:03.557487 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4" podUID="b40453ca-1141-43cc-a63c-63eae56fb4eb" containerName="payload-processing" containerID="cri-o://dcb9bc372ef94861e76880f095bd6c36ad651848891f4fd6e709fceceeaf8414" gracePeriod=30 Jul 30 15:46:03.811216 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:03.811147 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4" Jul 30 15:46:03.876399 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:03.876366 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvhzd\" (UniqueName: \"kubernetes.io/projected/b40453ca-1141-43cc-a63c-63eae56fb4eb-kube-api-access-xvhzd\") pod \"b40453ca-1141-43cc-a63c-63eae56fb4eb\" (UID: \"b40453ca-1141-43cc-a63c-63eae56fb4eb\") " Jul 30 15:46:03.876560 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:03.876479 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40453ca-1141-43cc-a63c-63eae56fb4eb-plugins-config-volume\") pod \"b40453ca-1141-43cc-a63c-63eae56fb4eb\" (UID: \"b40453ca-1141-43cc-a63c-63eae56fb4eb\") " Jul 30 15:46:03.876896 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:03.876861 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40453ca-1141-43cc-a63c-63eae56fb4eb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b40453ca-1141-43cc-a63c-63eae56fb4eb" (UID: "b40453ca-1141-43cc-a63c-63eae56fb4eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:46:03.878360 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:03.878334 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40453ca-1141-43cc-a63c-63eae56fb4eb-kube-api-access-xvhzd" (OuterVolumeSpecName: "kube-api-access-xvhzd") pod "b40453ca-1141-43cc-a63c-63eae56fb4eb" (UID: "b40453ca-1141-43cc-a63c-63eae56fb4eb"). InnerVolumeSpecName "kube-api-access-xvhzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:46:03.978311 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:03.978274 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvhzd\" (UniqueName: \"kubernetes.io/projected/b40453ca-1141-43cc-a63c-63eae56fb4eb-kube-api-access-xvhzd\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:46:03.978507 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:03.978353 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40453ca-1141-43cc-a63c-63eae56fb4eb-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:46:04.010422 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:04.010391 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc"] Jul 30 15:46:04.010641 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:04.010603 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" podUID="a254ff0f-3b60-44af-864d-25a8360adf3f" containerName="istio-proxy" containerID="cri-o://edb15e1aef54216b791c22b20ea0b08b6ffb00264b1ed93abeb86acc61df9c64" gracePeriod=30 Jul 30 15:46:04.267218 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:04.267187 2579 generic.go:356] "Generic (PLEG): container finished" podID="b40453ca-1141-43cc-a63c-63eae56fb4eb" containerID="dcb9bc372ef94861e76880f095bd6c36ad651848891f4fd6e709fceceeaf8414" exitCode=0 Jul 30 15:46:04.267410 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:04.267247 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4" Jul 30 15:46:04.267410 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:04.267276 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4" event={"ID":"b40453ca-1141-43cc-a63c-63eae56fb4eb","Type":"ContainerDied","Data":"dcb9bc372ef94861e76880f095bd6c36ad651848891f4fd6e709fceceeaf8414"} Jul 30 15:46:04.267410 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:04.267342 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4" event={"ID":"b40453ca-1141-43cc-a63c-63eae56fb4eb","Type":"ContainerDied","Data":"60eb0baf9e8905103cb1f125c793924169ea6776a0d1d6dc8ac2fd7630f966c6"} Jul 30 15:46:04.267410 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:04.267371 2579 scope.go:117] "RemoveContainer" containerID="dcb9bc372ef94861e76880f095bd6c36ad651848891f4fd6e709fceceeaf8414" Jul 30 15:46:04.276136 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:04.276119 2579 scope.go:117] "RemoveContainer" containerID="dcb9bc372ef94861e76880f095bd6c36ad651848891f4fd6e709fceceeaf8414" Jul 30 15:46:04.276411 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:46:04.276392 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcb9bc372ef94861e76880f095bd6c36ad651848891f4fd6e709fceceeaf8414\": container with ID starting with dcb9bc372ef94861e76880f095bd6c36ad651848891f4fd6e709fceceeaf8414 not found: ID does not exist" containerID="dcb9bc372ef94861e76880f095bd6c36ad651848891f4fd6e709fceceeaf8414" Jul 30 15:46:04.276483 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:04.276422 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcb9bc372ef94861e76880f095bd6c36ad651848891f4fd6e709fceceeaf8414"} err="failed to get container status \"dcb9bc372ef94861e76880f095bd6c36ad651848891f4fd6e709fceceeaf8414\": rpc error: code = NotFound desc = could not find container \"dcb9bc372ef94861e76880f095bd6c36ad651848891f4fd6e709fceceeaf8414\": container with ID starting with dcb9bc372ef94861e76880f095bd6c36ad651848891f4fd6e709fceceeaf8414 not found: ID does not exist" Jul 30 15:46:04.295599 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:04.295569 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4"] Jul 30 15:46:04.301699 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:04.301674 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d21ea6ee-5bb66bb7bb-p5pt4"] Jul 30 15:46:05.202465 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:05.202428 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b40453ca-1141-43cc-a63c-63eae56fb4eb" path="/var/lib/kubelet/pods/b40453ca-1141-43cc-a63c-63eae56fb4eb/volumes" Jul 30 15:46:06.722703 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:06.722660 2579 patch_prober.go:28] interesting pod/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 30 15:46:06.723106 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:06.722715 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" podUID="a254ff0f-3b60-44af-864d-25a8360adf3f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 15:46:07.766740 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:07.766708 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:46:09.265600 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.265577 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:46:09.296992 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.296928 2579 generic.go:356] "Generic (PLEG): container finished" podID="a254ff0f-3b60-44af-864d-25a8360adf3f" containerID="edb15e1aef54216b791c22b20ea0b08b6ffb00264b1ed93abeb86acc61df9c64" exitCode=0 Jul 30 15:46:09.296992 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.296972 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" Jul 30 15:46:09.297183 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.297011 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" event={"ID":"a254ff0f-3b60-44af-864d-25a8360adf3f","Type":"ContainerDied","Data":"edb15e1aef54216b791c22b20ea0b08b6ffb00264b1ed93abeb86acc61df9c64"} Jul 30 15:46:09.297183 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.297074 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc" event={"ID":"a254ff0f-3b60-44af-864d-25a8360adf3f","Type":"ContainerDied","Data":"a5560abb1418308e225cbfaa32b2a2ae457931819c4f0baec6bf4211ba795a2b"} Jul 30 15:46:09.297183 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.297106 2579 scope.go:117] "RemoveContainer" containerID="edb15e1aef54216b791c22b20ea0b08b6ffb00264b1ed93abeb86acc61df9c64" Jul 30 15:46:09.305639 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.305624 2579 scope.go:117] "RemoveContainer" containerID="22af1f77be97801152383869b18d49f78dea5d3258799776c12a09ac59fdd69a" Jul 30 15:46:09.313512 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.313490 2579 scope.go:117] "RemoveContainer" containerID="edb15e1aef54216b791c22b20ea0b08b6ffb00264b1ed93abeb86acc61df9c64" Jul 30 15:46:09.313773 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:46:09.313756 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb15e1aef54216b791c22b20ea0b08b6ffb00264b1ed93abeb86acc61df9c64\": container with ID starting with edb15e1aef54216b791c22b20ea0b08b6ffb00264b1ed93abeb86acc61df9c64 not found: ID does not exist" containerID="edb15e1aef54216b791c22b20ea0b08b6ffb00264b1ed93abeb86acc61df9c64" Jul 30 15:46:09.313882 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.313783 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb15e1aef54216b791c22b20ea0b08b6ffb00264b1ed93abeb86acc61df9c64"} err="failed to get container status \"edb15e1aef54216b791c22b20ea0b08b6ffb00264b1ed93abeb86acc61df9c64\": rpc error: code = NotFound desc = could not find container \"edb15e1aef54216b791c22b20ea0b08b6ffb00264b1ed93abeb86acc61df9c64\": container with ID starting with edb15e1aef54216b791c22b20ea0b08b6ffb00264b1ed93abeb86acc61df9c64 not found: ID does not exist" Jul 30 15:46:09.313882 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.313834 2579 scope.go:117] "RemoveContainer" containerID="22af1f77be97801152383869b18d49f78dea5d3258799776c12a09ac59fdd69a" Jul 30 15:46:09.314097 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:46:09.314077 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22af1f77be97801152383869b18d49f78dea5d3258799776c12a09ac59fdd69a\": container with ID starting with 22af1f77be97801152383869b18d49f78dea5d3258799776c12a09ac59fdd69a not found: ID does not exist" containerID="22af1f77be97801152383869b18d49f78dea5d3258799776c12a09ac59fdd69a" Jul 30 15:46:09.314148 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.314107 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22af1f77be97801152383869b18d49f78dea5d3258799776c12a09ac59fdd69a"} err="failed to get container status \"22af1f77be97801152383869b18d49f78dea5d3258799776c12a09ac59fdd69a\": rpc error: code = NotFound desc = could not find container \"22af1f77be97801152383869b18d49f78dea5d3258799776c12a09ac59fdd69a\": container with ID starting with 22af1f77be97801152383869b18d49f78dea5d3258799776c12a09ac59fdd69a not found: ID does not exist" Jul 30 15:46:09.330499 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.330479 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4d42\" (UniqueName: \"kubernetes.io/projected/a254ff0f-3b60-44af-864d-25a8360adf3f-kube-api-access-g4d42\") pod \"a254ff0f-3b60-44af-864d-25a8360adf3f\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " Jul 30 15:46:09.330600 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.330533 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-workload-socket\") pod \"a254ff0f-3b60-44af-864d-25a8360adf3f\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " Jul 30 15:46:09.330600 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.330559 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-credential-socket\") pod \"a254ff0f-3b60-44af-864d-25a8360adf3f\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " Jul 30 15:46:09.330712 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.330620 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-token\") pod \"a254ff0f-3b60-44af-864d-25a8360adf3f\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " Jul 30 15:46:09.330712 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.330672 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-podinfo\") pod \"a254ff0f-3b60-44af-864d-25a8360adf3f\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " Jul 30 15:46:09.330839 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.330714 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-envoy\") pod \"a254ff0f-3b60-44af-864d-25a8360adf3f\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " Jul 30 15:46:09.330839 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.330746 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a254ff0f-3b60-44af-864d-25a8360adf3f-istiod-ca-cert\") pod \"a254ff0f-3b60-44af-864d-25a8360adf3f\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " Jul 30 15:46:09.330947 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.330847 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-workload-certs\") pod \"a254ff0f-3b60-44af-864d-25a8360adf3f\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " Jul 30 15:46:09.330947 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.330859 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a254ff0f-3b60-44af-864d-25a8360adf3f" (UID: "a254ff0f-3b60-44af-864d-25a8360adf3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:46:09.330947 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.330874 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a254ff0f-3b60-44af-864d-25a8360adf3f" (UID: "a254ff0f-3b60-44af-864d-25a8360adf3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:46:09.330947 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.330913 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-data\") pod \"a254ff0f-3b60-44af-864d-25a8360adf3f\" (UID: \"a254ff0f-3b60-44af-864d-25a8360adf3f\") " Jul 30 15:46:09.331178 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.331157 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a254ff0f-3b60-44af-864d-25a8360adf3f" (UID: "a254ff0f-3b60-44af-864d-25a8360adf3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:46:09.331262 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.331194 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a254ff0f-3b60-44af-864d-25a8360adf3f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a254ff0f-3b60-44af-864d-25a8360adf3f" (UID: "a254ff0f-3b60-44af-864d-25a8360adf3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:46:09.331428 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.331406 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-workload-certs\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:46:09.331515 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.331432 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-workload-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:46:09.331515 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.331446 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-credential-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:46:09.331515 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.331461 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a254ff0f-3b60-44af-864d-25a8360adf3f-istiod-ca-cert\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:46:09.332175 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.332147 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-data" (OuterVolumeSpecName: "istio-data") pod "a254ff0f-3b60-44af-864d-25a8360adf3f" (UID: "a254ff0f-3b60-44af-864d-25a8360adf3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:46:09.333091 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.333064 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a254ff0f-3b60-44af-864d-25a8360adf3f" (UID: "a254ff0f-3b60-44af-864d-25a8360adf3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 15:46:09.333213 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.333187 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a254ff0f-3b60-44af-864d-25a8360adf3f-kube-api-access-g4d42" (OuterVolumeSpecName: "kube-api-access-g4d42") pod "a254ff0f-3b60-44af-864d-25a8360adf3f" (UID: "a254ff0f-3b60-44af-864d-25a8360adf3f"). InnerVolumeSpecName "kube-api-access-g4d42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:46:09.333309 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.333260 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a254ff0f-3b60-44af-864d-25a8360adf3f" (UID: "a254ff0f-3b60-44af-864d-25a8360adf3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:46:09.333309 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.333264 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-token" (OuterVolumeSpecName: "istio-token") pod "a254ff0f-3b60-44af-864d-25a8360adf3f" (UID: "a254ff0f-3b60-44af-864d-25a8360adf3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:46:09.432684 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.432656 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-envoy\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:46:09.432684 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.432681 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-data\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:46:09.432866 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.432691 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4d42\" (UniqueName: \"kubernetes.io/projected/a254ff0f-3b60-44af-864d-25a8360adf3f-kube-api-access-g4d42\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:46:09.432866 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.432700 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-token\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:46:09.432866 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.432710 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a254ff0f-3b60-44af-864d-25a8360adf3f-istio-podinfo\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:46:09.627093 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.627061 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc"] Jul 30 15:46:09.629548 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:09.629526 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-d21ea6ee-openshift-default-6db946587b-p59gc"] Jul 30 15:46:11.197694 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:11.197662 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a254ff0f-3b60-44af-864d-25a8360adf3f" path="/var/lib/kubelet/pods/a254ff0f-3b60-44af-864d-25a8360adf3f/volumes" Jul 30 15:46:30.549194 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:46:30.549159 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:48:08.998651 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:08.998618 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:48:21.948760 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:21.948720 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:48:24.816876 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.816846 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m"] Jul 30 15:48:24.817331 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.817314 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a254ff0f-3b60-44af-864d-25a8360adf3f" containerName="istio-proxy" Jul 30 15:48:24.817331 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.817330 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a254ff0f-3b60-44af-864d-25a8360adf3f" containerName="istio-proxy" Jul 30 15:48:24.817446 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.817354 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94c707ff-3887-43c1-9165-1fdb05de2fbb" containerName="istio-proxy" Jul 30 15:48:24.817446 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.817361 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="94c707ff-3887-43c1-9165-1fdb05de2fbb" containerName="istio-proxy" Jul 30 15:48:24.817446 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.817388 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b40453ca-1141-43cc-a63c-63eae56fb4eb" containerName="payload-processing" Jul 30 15:48:24.817446 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.817397 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b40453ca-1141-43cc-a63c-63eae56fb4eb" containerName="payload-processing" Jul 30 15:48:24.817580 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.817463 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="94c707ff-3887-43c1-9165-1fdb05de2fbb" containerName="istio-proxy" Jul 30 15:48:24.817580 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.817472 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a254ff0f-3b60-44af-864d-25a8360adf3f" containerName="istio-proxy" Jul 30 15:48:24.817580 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.817479 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a254ff0f-3b60-44af-864d-25a8360adf3f" containerName="istio-proxy" Jul 30 15:48:24.817580 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.817487 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b40453ca-1141-43cc-a63c-63eae56fb4eb" containerName="payload-processing" Jul 30 15:48:24.820484 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.820470 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:24.824395 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.824374 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-44cb5ac4-openshift-default-dockercfg-56rvk\"" Jul 30 15:48:24.864810 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.864769 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m"] Jul 30 15:48:24.911077 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.911042 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-workload-socket\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:24.911077 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.911081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istiod-ca-cert\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:24.911272 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.911128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-token\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:24.911272 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.911158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-credential-socket\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:24.911272 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.911179 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-envoy\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:24.911272 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.911206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-podinfo\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:24.911272 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.911237 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-data\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:24.911272 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.911255 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-workload-certs\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:24.911463 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:24.911276 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqg5w\" (UniqueName: \"kubernetes.io/projected/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-kube-api-access-bqg5w\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.012193 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.012156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-data\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.012375 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.012198 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-workload-certs\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.012375 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.012222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqg5w\" (UniqueName: \"kubernetes.io/projected/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-kube-api-access-bqg5w\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.012375 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.012288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-workload-socket\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.012375 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.012324 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istiod-ca-cert\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.012375 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.012369 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-token\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.012653 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.012411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-credential-socket\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.012653 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.012444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-envoy\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.012653 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.012479 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-podinfo\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.012653 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.012608 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-data\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.012870 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.012652 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-workload-certs\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.012870 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.012698 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-workload-socket\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.012936 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.012895 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-credential-socket\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.013124 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.013101 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istiod-ca-cert\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.014851 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.014831 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-envoy\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.015008 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.014991 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-podinfo\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.024196 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.024176 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-token\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.024705 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.024686 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqg5w\" (UniqueName: \"kubernetes.io/projected/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-kube-api-access-bqg5w\") pod \"e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.130558 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.130530 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:25.272679 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.272653 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m"] Jul 30 15:48:25.274197 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:48:25.274173 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c5ce22_79aa_4c9f_b13e_1ef6d7f65ec2.slice/crio-613885913f43829ba0f13e19f0e6f4ac1a81b5d1e945cdf4cdd688b4c15c8b38 WatchSource:0}: Error finding container 613885913f43829ba0f13e19f0e6f4ac1a81b5d1e945cdf4cdd688b4c15c8b38: Status 404 returned error can't find the container with id 613885913f43829ba0f13e19f0e6f4ac1a81b5d1e945cdf4cdd688b4c15c8b38 Jul 30 15:48:25.276302 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.276271 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:48:25.276377 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.276333 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:48:25.276377 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.276365 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:48:25.960350 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.960314 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" event={"ID":"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2","Type":"ContainerStarted","Data":"45d06ad0a1853db0ba37fcd6115a632aca1ad3eb333f35a9c4879f0d4e2af0aa"} Jul 30 15:48:25.960849 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.960364 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" event={"ID":"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2","Type":"ContainerStarted","Data":"613885913f43829ba0f13e19f0e6f4ac1a81b5d1e945cdf4cdd688b4c15c8b38"} Jul 30 15:48:25.985684 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:25.985633 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" podStartSLOduration=1.985617393 podStartE2EDuration="1.985617393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:48:25.983669883 +0000 UTC m=+2333.218800730" watchObservedRunningTime="2026-07-30 15:48:25.985617393 +0000 UTC m=+2333.220748236" Jul 30 15:48:26.131284 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:26.131252 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:26.135576 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:26.135552 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:26.965134 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:26.965096 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:26.965947 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:26.965925 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:29.689778 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.689746 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl"] Jul 30 15:48:29.690263 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.690141 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a254ff0f-3b60-44af-864d-25a8360adf3f" containerName="istio-proxy" Jul 30 15:48:29.690263 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.690153 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a254ff0f-3b60-44af-864d-25a8360adf3f" containerName="istio-proxy" Jul 30 15:48:29.690263 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.690168 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94c707ff-3887-43c1-9165-1fdb05de2fbb" containerName="istio-proxy" Jul 30 15:48:29.690263 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.690174 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="94c707ff-3887-43c1-9165-1fdb05de2fbb" containerName="istio-proxy" Jul 30 15:48:29.690263 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.690237 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="94c707ff-3887-43c1-9165-1fdb05de2fbb" containerName="istio-proxy" Jul 30 15:48:29.693404 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.693387 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl" Jul 30 15:48:29.696177 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.696158 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-44cb5ac4\"" Jul 30 15:48:29.705381 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.705360 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl"] Jul 30 15:48:29.727016 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.726994 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx"] Jul 30 15:48:29.730246 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.730231 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx" Jul 30 15:48:29.733218 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.733201 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-44cb5ac4-dockercfg-nmwjm\"" Jul 30 15:48:29.755699 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.755670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfj7d\" (UniqueName: \"kubernetes.io/projected/431ea2cf-d6d7-4833-b6c3-0320c01c6141-kube-api-access-rfj7d\") pod \"payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl\" (UID: \"431ea2cf-d6d7-4833-b6c3-0320c01c6141\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl" Jul 30 15:48:29.755842 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.755707 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzl86\" (UniqueName: \"kubernetes.io/projected/a173d667-0387-4668-af66-208bffff81d4-kube-api-access-vzl86\") pod \"payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx\" (UID: \"a173d667-0387-4668-af66-208bffff81d4\") " pod="openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx" Jul 30 15:48:29.755842 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.755738 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a173d667-0387-4668-af66-208bffff81d4-plugins-config-volume\") pod \"payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx\" (UID: \"a173d667-0387-4668-af66-208bffff81d4\") " pod="openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx" Jul 30 15:48:29.755842 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.755825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/431ea2cf-d6d7-4833-b6c3-0320c01c6141-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl\" (UID: \"431ea2cf-d6d7-4833-b6c3-0320c01c6141\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl" Jul 30 15:48:29.757430 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.757411 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx"] Jul 30 15:48:29.856834 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.856780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfj7d\" (UniqueName: \"kubernetes.io/projected/431ea2cf-d6d7-4833-b6c3-0320c01c6141-kube-api-access-rfj7d\") pod \"payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl\" (UID: \"431ea2cf-d6d7-4833-b6c3-0320c01c6141\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl" Jul 30 15:48:29.857044 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.856842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzl86\" (UniqueName: \"kubernetes.io/projected/a173d667-0387-4668-af66-208bffff81d4-kube-api-access-vzl86\") pod \"payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx\" (UID: \"a173d667-0387-4668-af66-208bffff81d4\") " pod="openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx" Jul 30 15:48:29.857044 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.856972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a173d667-0387-4668-af66-208bffff81d4-plugins-config-volume\") pod \"payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx\" (UID: \"a173d667-0387-4668-af66-208bffff81d4\") " pod="openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx" Jul 30 15:48:29.857147 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.857050 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/431ea2cf-d6d7-4833-b6c3-0320c01c6141-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl\" (UID: \"431ea2cf-d6d7-4833-b6c3-0320c01c6141\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl" Jul 30 15:48:29.857684 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.857666 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a173d667-0387-4668-af66-208bffff81d4-plugins-config-volume\") pod \"payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx\" (UID: \"a173d667-0387-4668-af66-208bffff81d4\") " pod="openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx" Jul 30 15:48:29.857726 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.857707 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/431ea2cf-d6d7-4833-b6c3-0320c01c6141-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl\" (UID: \"431ea2cf-d6d7-4833-b6c3-0320c01c6141\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl" Jul 30 15:48:29.866342 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.866313 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzl86\" (UniqueName: \"kubernetes.io/projected/a173d667-0387-4668-af66-208bffff81d4-kube-api-access-vzl86\") pod \"payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx\" (UID: \"a173d667-0387-4668-af66-208bffff81d4\") " pod="openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx" Jul 30 15:48:29.866441 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:29.866427 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfj7d\" (UniqueName: \"kubernetes.io/projected/431ea2cf-d6d7-4833-b6c3-0320c01c6141-kube-api-access-rfj7d\") pod \"payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl\" (UID: \"431ea2cf-d6d7-4833-b6c3-0320c01c6141\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl" Jul 30 15:48:30.005265 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:30.005173 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl" Jul 30 15:48:30.039705 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:30.039673 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx" Jul 30 15:48:30.153430 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:30.153398 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl"] Jul 30 15:48:30.154633 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:48:30.154588 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431ea2cf_d6d7_4833_b6c3_0320c01c6141.slice/crio-8dfd99006be5bb0dafdb00a46c27ae21a88e1230b17bc7ca7e770c82e7ec8f32 WatchSource:0}: Error finding container 8dfd99006be5bb0dafdb00a46c27ae21a88e1230b17bc7ca7e770c82e7ec8f32: Status 404 returned error can't find the container with id 8dfd99006be5bb0dafdb00a46c27ae21a88e1230b17bc7ca7e770c82e7ec8f32 Jul 30 15:48:30.195455 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:30.195416 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx"] Jul 30 15:48:30.196852 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:48:30.196827 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda173d667_0387_4668_af66_208bffff81d4.slice/crio-d9e1ce8c9ef0622ecd69af5014c5e412314ce70af2d83d16a540c64fe916a853 WatchSource:0}: Error finding container d9e1ce8c9ef0622ecd69af5014c5e412314ce70af2d83d16a540c64fe916a853: Status 404 returned error can't find the container with id d9e1ce8c9ef0622ecd69af5014c5e412314ce70af2d83d16a540c64fe916a853 Jul 30 15:48:30.987755 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:30.987718 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx" event={"ID":"a173d667-0387-4668-af66-208bffff81d4","Type":"ContainerStarted","Data":"3e9d3bf62fbd7d908cd1ed87649684e7187c0faf657559735fbbaf0778d8a3be"} Jul 30 15:48:30.988271 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:30.987897 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx" Jul 30 15:48:30.988271 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:30.987945 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx" event={"ID":"a173d667-0387-4668-af66-208bffff81d4","Type":"ContainerStarted","Data":"d9e1ce8c9ef0622ecd69af5014c5e412314ce70af2d83d16a540c64fe916a853"} Jul 30 15:48:30.989192 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:30.989172 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl" event={"ID":"431ea2cf-d6d7-4833-b6c3-0320c01c6141","Type":"ContainerStarted","Data":"61d02dd42740e00c31ba330c2286c32fb92b18e0c163e7120466fb1cffd42407"} Jul 30 15:48:30.989286 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:30.989205 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl" event={"ID":"431ea2cf-d6d7-4833-b6c3-0320c01c6141","Type":"ContainerStarted","Data":"8dfd99006be5bb0dafdb00a46c27ae21a88e1230b17bc7ca7e770c82e7ec8f32"} Jul 30 15:48:30.989286 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:30.989232 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl" Jul 30 15:48:31.010386 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:31.010343 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx" podStartSLOduration=2.010329696 podStartE2EDuration="2.010329696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:48:31.009072039 +0000 UTC m=+2338.244202897" watchObservedRunningTime="2026-07-30 15:48:31.010329696 +0000 UTC m=+2338.245460541" Jul 30 15:48:31.027719 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:31.027683 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl" podStartSLOduration=2.02766999 podStartE2EDuration="2.02766999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:48:31.026934055 +0000 UTC m=+2338.262064904" watchObservedRunningTime="2026-07-30 15:48:31.02766999 +0000 UTC m=+2338.262800835" Jul 30 15:48:35.160210 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:35.160177 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:48:41.996132 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:41.996067 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx" Jul 30 15:48:41.996524 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:41.996464 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl" Jul 30 15:48:43.005956 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:43.005921 2579 patch_prober.go:28] interesting pod/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 30 15:48:43.007483 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:43.006016 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" podUID="52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 30 15:48:43.053726 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:43.053701 2579 generic.go:356] "Generic (PLEG): container finished" podID="52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" containerID="45d06ad0a1853db0ba37fcd6115a632aca1ad3eb333f35a9c4879f0d4e2af0aa" exitCode=0 Jul 30 15:48:43.053845 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:43.053769 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" event={"ID":"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2","Type":"ContainerDied","Data":"45d06ad0a1853db0ba37fcd6115a632aca1ad3eb333f35a9c4879f0d4e2af0aa"} Jul 30 15:48:43.054179 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:43.054165 2579 scope.go:117] "RemoveContainer" containerID="45d06ad0a1853db0ba37fcd6115a632aca1ad3eb333f35a9c4879f0d4e2af0aa" Jul 30 15:48:43.055143 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:43.055113 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:48:43.055221 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:43.055204 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:48:43.055258 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:43.055245 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:48:44.064387 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:44.064353 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" event={"ID":"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2","Type":"ContainerStarted","Data":"355d8fb1fe4895926c6ede74a35f95bf8581d3fa27ab49342e46c95654c58a25"} Jul 30 15:48:44.131030 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:44.130997 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:44.131303 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:44.131277 2579 patch_prober.go:28] interesting pod/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 30 15:48:44.131357 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:44.131324 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" podUID="52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 30 15:48:45.131510 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:45.131472 2579 patch_prober.go:28] interesting pod/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 30 15:48:45.131947 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:45.131530 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" podUID="52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 30 15:48:46.130877 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:46.130839 2579 patch_prober.go:28] interesting pod/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 30 15:48:46.131085 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:46.130894 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" podUID="52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 30 15:48:47.131865 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:47.131827 2579 patch_prober.go:28] interesting pod/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 30 15:48:47.132322 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:47.131889 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" podUID="52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 30 15:48:48.134968 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:48.134938 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:48.135374 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:48.135206 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:48:48.135952 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:48:48.135935 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:49:11.523143 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:11.523110 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx"] Jul 30 15:49:11.523728 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:11.523390 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx" podUID="a173d667-0387-4668-af66-208bffff81d4" containerName="payload-processing" containerID="cri-o://3e9d3bf62fbd7d908cd1ed87649684e7187c0faf657559735fbbaf0778d8a3be" gracePeriod=30 Jul 30 15:49:11.526942 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:11.526912 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl"] Jul 30 15:49:11.527849 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:11.527778 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl" podUID="431ea2cf-d6d7-4833-b6c3-0320c01c6141" containerName="payload-pre-processing" containerID="cri-o://61d02dd42740e00c31ba330c2286c32fb92b18e0c163e7120466fb1cffd42407" gracePeriod=30 Jul 30 15:49:11.797462 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:11.797437 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl" Jul 30 15:49:11.812251 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:11.812231 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx" Jul 30 15:49:11.836295 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:11.836268 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/431ea2cf-d6d7-4833-b6c3-0320c01c6141-plugins-config-volume\") pod \"431ea2cf-d6d7-4833-b6c3-0320c01c6141\" (UID: \"431ea2cf-d6d7-4833-b6c3-0320c01c6141\") " Jul 30 15:49:11.836393 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:11.836334 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzl86\" (UniqueName: \"kubernetes.io/projected/a173d667-0387-4668-af66-208bffff81d4-kube-api-access-vzl86\") pod \"a173d667-0387-4668-af66-208bffff81d4\" (UID: \"a173d667-0387-4668-af66-208bffff81d4\") " Jul 30 15:49:11.836393 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:11.836379 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a173d667-0387-4668-af66-208bffff81d4-plugins-config-volume\") pod \"a173d667-0387-4668-af66-208bffff81d4\" (UID: \"a173d667-0387-4668-af66-208bffff81d4\") " Jul 30 15:49:11.836463 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:11.836452 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfj7d\" (UniqueName: \"kubernetes.io/projected/431ea2cf-d6d7-4833-b6c3-0320c01c6141-kube-api-access-rfj7d\") pod \"431ea2cf-d6d7-4833-b6c3-0320c01c6141\" (UID: \"431ea2cf-d6d7-4833-b6c3-0320c01c6141\") " Jul 30 15:49:11.836756 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:11.836733 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431ea2cf-d6d7-4833-b6c3-0320c01c6141-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "431ea2cf-d6d7-4833-b6c3-0320c01c6141" (UID: "431ea2cf-d6d7-4833-b6c3-0320c01c6141"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:49:11.836856 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:11.836756 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a173d667-0387-4668-af66-208bffff81d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a173d667-0387-4668-af66-208bffff81d4" (UID: "a173d667-0387-4668-af66-208bffff81d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:49:11.838586 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:11.838556 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431ea2cf-d6d7-4833-b6c3-0320c01c6141-kube-api-access-rfj7d" (OuterVolumeSpecName: "kube-api-access-rfj7d") pod "431ea2cf-d6d7-4833-b6c3-0320c01c6141" (UID: "431ea2cf-d6d7-4833-b6c3-0320c01c6141"). InnerVolumeSpecName "kube-api-access-rfj7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:49:11.838694 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:11.838661 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a173d667-0387-4668-af66-208bffff81d4-kube-api-access-vzl86" (OuterVolumeSpecName: "kube-api-access-vzl86") pod "a173d667-0387-4668-af66-208bffff81d4" (UID: "a173d667-0387-4668-af66-208bffff81d4"). InnerVolumeSpecName "kube-api-access-vzl86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:49:11.937774 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:11.937740 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzl86\" (UniqueName: \"kubernetes.io/projected/a173d667-0387-4668-af66-208bffff81d4-kube-api-access-vzl86\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:49:11.937774 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:11.937768 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a173d667-0387-4668-af66-208bffff81d4-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:49:11.937774 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:11.937778 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfj7d\" (UniqueName: \"kubernetes.io/projected/431ea2cf-d6d7-4833-b6c3-0320c01c6141-kube-api-access-rfj7d\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:49:11.938008 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:11.937786 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/431ea2cf-d6d7-4833-b6c3-0320c01c6141-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:49:12.202149 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:12.202113 2579 generic.go:356] "Generic (PLEG): container finished" podID="431ea2cf-d6d7-4833-b6c3-0320c01c6141" containerID="61d02dd42740e00c31ba330c2286c32fb92b18e0c163e7120466fb1cffd42407" exitCode=0 Jul 30 15:49:12.202149 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:12.202148 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl" Jul 30 15:49:12.202383 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:12.202198 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl" event={"ID":"431ea2cf-d6d7-4833-b6c3-0320c01c6141","Type":"ContainerDied","Data":"61d02dd42740e00c31ba330c2286c32fb92b18e0c163e7120466fb1cffd42407"} Jul 30 15:49:12.202383 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:12.202243 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl" event={"ID":"431ea2cf-d6d7-4833-b6c3-0320c01c6141","Type":"ContainerDied","Data":"8dfd99006be5bb0dafdb00a46c27ae21a88e1230b17bc7ca7e770c82e7ec8f32"} Jul 30 15:49:12.202383 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:12.202270 2579 scope.go:117] "RemoveContainer" containerID="61d02dd42740e00c31ba330c2286c32fb92b18e0c163e7120466fb1cffd42407" Jul 30 15:49:12.203591 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:12.203561 2579 generic.go:356] "Generic (PLEG): container finished" podID="a173d667-0387-4668-af66-208bffff81d4" containerID="3e9d3bf62fbd7d908cd1ed87649684e7187c0faf657559735fbbaf0778d8a3be" exitCode=0 Jul 30 15:49:12.203729 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:12.203607 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx" Jul 30 15:49:12.203729 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:12.203635 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx" event={"ID":"a173d667-0387-4668-af66-208bffff81d4","Type":"ContainerDied","Data":"3e9d3bf62fbd7d908cd1ed87649684e7187c0faf657559735fbbaf0778d8a3be"} Jul 30 15:49:12.203729 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:12.203687 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx" event={"ID":"a173d667-0387-4668-af66-208bffff81d4","Type":"ContainerDied","Data":"d9e1ce8c9ef0622ecd69af5014c5e412314ce70af2d83d16a540c64fe916a853"} Jul 30 15:49:12.211508 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:12.211382 2579 scope.go:117] "RemoveContainer" containerID="61d02dd42740e00c31ba330c2286c32fb92b18e0c163e7120466fb1cffd42407" Jul 30 15:49:12.211741 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:49:12.211709 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61d02dd42740e00c31ba330c2286c32fb92b18e0c163e7120466fb1cffd42407\": container with ID starting with 61d02dd42740e00c31ba330c2286c32fb92b18e0c163e7120466fb1cffd42407 not found: ID does not exist" containerID="61d02dd42740e00c31ba330c2286c32fb92b18e0c163e7120466fb1cffd42407" Jul 30 15:49:12.211861 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:12.211744 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61d02dd42740e00c31ba330c2286c32fb92b18e0c163e7120466fb1cffd42407"} err="failed to get container status \"61d02dd42740e00c31ba330c2286c32fb92b18e0c163e7120466fb1cffd42407\": rpc error: code = NotFound desc = could not find container \"61d02dd42740e00c31ba330c2286c32fb92b18e0c163e7120466fb1cffd42407\": container with ID starting with 61d02dd42740e00c31ba330c2286c32fb92b18e0c163e7120466fb1cffd42407 not found: ID does not exist" Jul 30 15:49:12.211861 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:12.211770 2579 scope.go:117] "RemoveContainer" containerID="3e9d3bf62fbd7d908cd1ed87649684e7187c0faf657559735fbbaf0778d8a3be" Jul 30 15:49:12.219602 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:12.219586 2579 scope.go:117] "RemoveContainer" containerID="3e9d3bf62fbd7d908cd1ed87649684e7187c0faf657559735fbbaf0778d8a3be" Jul 30 15:49:12.219881 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:49:12.219859 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9d3bf62fbd7d908cd1ed87649684e7187c0faf657559735fbbaf0778d8a3be\": container with ID starting with 3e9d3bf62fbd7d908cd1ed87649684e7187c0faf657559735fbbaf0778d8a3be not found: ID does not exist" containerID="3e9d3bf62fbd7d908cd1ed87649684e7187c0faf657559735fbbaf0778d8a3be" Jul 30 15:49:12.219954 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:12.219889 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9d3bf62fbd7d908cd1ed87649684e7187c0faf657559735fbbaf0778d8a3be"} err="failed to get container status \"3e9d3bf62fbd7d908cd1ed87649684e7187c0faf657559735fbbaf0778d8a3be\": rpc error: code = NotFound desc = could not find container \"3e9d3bf62fbd7d908cd1ed87649684e7187c0faf657559735fbbaf0778d8a3be\": container with ID starting with 3e9d3bf62fbd7d908cd1ed87649684e7187c0faf657559735fbbaf0778d8a3be not found: ID does not exist" Jul 30 15:49:12.230065 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:12.230042 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl"] Jul 30 15:49:12.240103 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:12.240081 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-44cb5ac4-7dccd5984-p97rl"] Jul 30 15:49:12.256329 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:12.256312 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx"] Jul 30 15:49:12.266344 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:12.266327 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-44cb5ac4-5cf74bb46-cn9dx"] Jul 30 15:49:13.202996 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:13.202958 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="431ea2cf-d6d7-4833-b6c3-0320c01c6141" path="/var/lib/kubelet/pods/431ea2cf-d6d7-4833-b6c3-0320c01c6141/volumes" Jul 30 15:49:13.203507 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:13.203486 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a173d667-0387-4668-af66-208bffff81d4" path="/var/lib/kubelet/pods/a173d667-0387-4668-af66-208bffff81d4/volumes" Jul 30 15:49:14.250834 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:14.250775 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:49:19.421326 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:19.421297 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m"] Jul 30 15:49:19.421739 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:19.421531 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" podUID="52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" containerName="istio-proxy" containerID="cri-o://355d8fb1fe4895926c6ede74a35f95bf8581d3fa27ab49342e46c95654c58a25" gracePeriod=30 Jul 30 15:49:24.671179 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.671149 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:49:24.742588 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.742525 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-data\") pod \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " Jul 30 15:49:24.742588 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.742562 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-envoy\") pod \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " Jul 30 15:49:24.742792 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.742617 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-workload-socket\") pod \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " Jul 30 15:49:24.742792 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.742678 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-credential-socket\") pod \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " Jul 30 15:49:24.742792 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.742706 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-podinfo\") pod \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " Jul 30 15:49:24.742792 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.742737 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-workload-certs\") pod \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " Jul 30 15:49:24.743040 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.742829 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-token\") pod \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " Jul 30 15:49:24.743040 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.742861 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istiod-ca-cert\") pod \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " Jul 30 15:49:24.743040 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.742895 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqg5w\" (UniqueName: \"kubernetes.io/projected/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-kube-api-access-bqg5w\") pod \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\" (UID: \"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2\") " Jul 30 15:49:24.743209 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.743141 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" (UID: "52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:49:24.743268 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.743219 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" (UID: "52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:49:24.743268 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.743246 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" (UID: "52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:49:24.743628 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.743582 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-data" (OuterVolumeSpecName: "istio-data") pod "52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" (UID: "52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:49:24.743628 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.743620 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-workload-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:49:24.743826 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.743655 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-credential-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:49:24.743826 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.743710 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" (UID: "52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:49:24.745171 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.745150 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" (UID: "52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:49:24.745566 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.745535 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-token" (OuterVolumeSpecName: "istio-token") pod "52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" (UID: "52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:49:24.745679 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.745571 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" (UID: "52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 15:49:24.745679 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.745632 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-kube-api-access-bqg5w" (OuterVolumeSpecName: "kube-api-access-bqg5w") pod "52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" (UID: "52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2"). InnerVolumeSpecName "kube-api-access-bqg5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:49:24.845076 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.845053 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-token\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:49:24.845076 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.845075 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istiod-ca-cert\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:49:24.845233 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.845084 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqg5w\" (UniqueName: \"kubernetes.io/projected/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-kube-api-access-bqg5w\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:49:24.845233 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.845099 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-data\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:49:24.845233 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.845108 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-envoy\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:49:24.845233 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.845116 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-istio-podinfo\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:49:24.845233 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:24.845123 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2-workload-certs\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:49:25.268402 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:25.268370 2579 generic.go:356] "Generic (PLEG): container finished" podID="52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" containerID="355d8fb1fe4895926c6ede74a35f95bf8581d3fa27ab49342e46c95654c58a25" exitCode=0 Jul 30 15:49:25.268529 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:25.268414 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" event={"ID":"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2","Type":"ContainerDied","Data":"355d8fb1fe4895926c6ede74a35f95bf8581d3fa27ab49342e46c95654c58a25"} Jul 30 15:49:25.268529 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:25.268422 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" Jul 30 15:49:25.268529 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:25.268448 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m" event={"ID":"52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2","Type":"ContainerDied","Data":"613885913f43829ba0f13e19f0e6f4ac1a81b5d1e945cdf4cdd688b4c15c8b38"} Jul 30 15:49:25.268529 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:25.268468 2579 scope.go:117] "RemoveContainer" containerID="355d8fb1fe4895926c6ede74a35f95bf8581d3fa27ab49342e46c95654c58a25" Jul 30 15:49:25.277571 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:25.277556 2579 scope.go:117] "RemoveContainer" containerID="45d06ad0a1853db0ba37fcd6115a632aca1ad3eb333f35a9c4879f0d4e2af0aa" Jul 30 15:49:25.285127 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:25.285105 2579 scope.go:117] "RemoveContainer" containerID="355d8fb1fe4895926c6ede74a35f95bf8581d3fa27ab49342e46c95654c58a25" Jul 30 15:49:25.285373 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:49:25.285344 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355d8fb1fe4895926c6ede74a35f95bf8581d3fa27ab49342e46c95654c58a25\": container with ID starting with 355d8fb1fe4895926c6ede74a35f95bf8581d3fa27ab49342e46c95654c58a25 not found: ID does not exist" containerID="355d8fb1fe4895926c6ede74a35f95bf8581d3fa27ab49342e46c95654c58a25" Jul 30 15:49:25.285448 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:25.285385 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355d8fb1fe4895926c6ede74a35f95bf8581d3fa27ab49342e46c95654c58a25"} err="failed to get container status \"355d8fb1fe4895926c6ede74a35f95bf8581d3fa27ab49342e46c95654c58a25\": rpc error: code = NotFound desc = could not find container \"355d8fb1fe4895926c6ede74a35f95bf8581d3fa27ab49342e46c95654c58a25\": container with ID starting with 355d8fb1fe4895926c6ede74a35f95bf8581d3fa27ab49342e46c95654c58a25 not found: ID does not exist" Jul 30 15:49:25.285448 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:25.285407 2579 scope.go:117] "RemoveContainer" containerID="45d06ad0a1853db0ba37fcd6115a632aca1ad3eb333f35a9c4879f0d4e2af0aa" Jul 30 15:49:25.285652 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:49:25.285633 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d06ad0a1853db0ba37fcd6115a632aca1ad3eb333f35a9c4879f0d4e2af0aa\": container with ID starting with 45d06ad0a1853db0ba37fcd6115a632aca1ad3eb333f35a9c4879f0d4e2af0aa not found: ID does not exist" containerID="45d06ad0a1853db0ba37fcd6115a632aca1ad3eb333f35a9c4879f0d4e2af0aa" Jul 30 15:49:25.285695 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:25.285657 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d06ad0a1853db0ba37fcd6115a632aca1ad3eb333f35a9c4879f0d4e2af0aa"} err="failed to get container status \"45d06ad0a1853db0ba37fcd6115a632aca1ad3eb333f35a9c4879f0d4e2af0aa\": rpc error: code = NotFound desc = could not find container \"45d06ad0a1853db0ba37fcd6115a632aca1ad3eb333f35a9c4879f0d4e2af0aa\": container with ID starting with 45d06ad0a1853db0ba37fcd6115a632aca1ad3eb333f35a9c4879f0d4e2af0aa not found: ID does not exist" Jul 30 15:49:25.296005 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:25.295983 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m"] Jul 30 15:49:25.299558 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:25.299534 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-44cb5ac4-openshift-default-687dd9d5b4-nwr7m"] Jul 30 15:49:27.197291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:27.197254 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" path="/var/lib/kubelet/pods/52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2/volumes" Jul 30 15:49:57.350291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:49:57.350210 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:50:33.472520 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:50:33.472483 2579 scope.go:117] "RemoveContainer" containerID="5be9d2bd90ccd07888acce30bfa7d319c7072887b4950d779ead02dafee2866e" Jul 30 15:51:47.855609 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.855516 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j"] Jul 30 15:51:47.856291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.855991 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="431ea2cf-d6d7-4833-b6c3-0320c01c6141" containerName="payload-pre-processing" Jul 30 15:51:47.856291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.856004 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="431ea2cf-d6d7-4833-b6c3-0320c01c6141" containerName="payload-pre-processing" Jul 30 15:51:47.856291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.856018 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" containerName="istio-proxy" Jul 30 15:51:47.856291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.856024 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" containerName="istio-proxy" Jul 30 15:51:47.856291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.856036 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" containerName="istio-proxy" Jul 30 15:51:47.856291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.856041 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" containerName="istio-proxy" Jul 30 15:51:47.856291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.856055 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a173d667-0387-4668-af66-208bffff81d4" containerName="payload-processing" Jul 30 15:51:47.856291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.856060 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a173d667-0387-4668-af66-208bffff81d4" containerName="payload-processing" Jul 30 15:51:47.856291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.856121 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a173d667-0387-4668-af66-208bffff81d4" containerName="payload-processing" Jul 30 15:51:47.856291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.856129 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="431ea2cf-d6d7-4833-b6c3-0320c01c6141" containerName="payload-pre-processing" Jul 30 15:51:47.856291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.856136 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" containerName="istio-proxy" Jul 30 15:51:47.856291 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.856143 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="52c5ce22-79aa-4c9f-b13e-1ef6d7f65ec2" containerName="istio-proxy" Jul 30 15:51:47.859416 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.859399 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:47.863111 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.863093 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-db19ef93-openshift-default-dockercfg-tlf5x\"" Jul 30 15:51:47.875355 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.875333 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j"] Jul 30 15:51:47.911027 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.910997 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-credential-socket\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:47.911152 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.911034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-workload-socket\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:47.911152 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.911052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f8xv\" (UniqueName: \"kubernetes.io/projected/d214c878-41c3-437e-ab93-117823bcc7fa-kube-api-access-9f8xv\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:47.911256 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.911156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d214c878-41c3-437e-ab93-117823bcc7fa-istio-token\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:47.911256 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.911207 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-workload-certs\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:47.911336 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.911275 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d214c878-41c3-437e-ab93-117823bcc7fa-istio-podinfo\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:47.911336 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.911300 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-istio-envoy\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:47.911336 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.911319 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d214c878-41c3-437e-ab93-117823bcc7fa-istiod-ca-cert\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:47.911336 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:47.911334 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-istio-data\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.012617 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.012581 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-istio-envoy\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.012617 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.012620 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d214c878-41c3-437e-ab93-117823bcc7fa-istiod-ca-cert\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.012922 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.012637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-istio-data\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.012922 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.012666 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-credential-socket\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.012922 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.012686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-workload-socket\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.012922 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.012706 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f8xv\" (UniqueName: \"kubernetes.io/projected/d214c878-41c3-437e-ab93-117823bcc7fa-kube-api-access-9f8xv\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.012922 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.012737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d214c878-41c3-437e-ab93-117823bcc7fa-istio-token\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.012922 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.012786 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-workload-certs\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.012922 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.012893 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d214c878-41c3-437e-ab93-117823bcc7fa-istio-podinfo\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.013303 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.013076 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-istio-data\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.013391 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.013351 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-workload-socket\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.013455 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.013374 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-credential-socket\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.013455 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.013436 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-workload-certs\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.013549 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.013442 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d214c878-41c3-437e-ab93-117823bcc7fa-istiod-ca-cert\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.014968 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.014948 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-istio-envoy\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.015239 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.015222 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d214c878-41c3-437e-ab93-117823bcc7fa-istio-podinfo\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.023062 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.023041 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f8xv\" (UniqueName: \"kubernetes.io/projected/d214c878-41c3-437e-ab93-117823bcc7fa-kube-api-access-9f8xv\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.026383 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.026364 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d214c878-41c3-437e-ab93-117823bcc7fa-istio-token\") pod \"e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.169418 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.169392 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:48.309623 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.309598 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j"] Jul 30 15:51:48.311636 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:51:48.311606 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd214c878_41c3_437e_ab93_117823bcc7fa.slice/crio-0f06b7a72703424c818563040dadf58db909ed3598019fa55ca9fdf44aadab80 WatchSource:0}: Error finding container 0f06b7a72703424c818563040dadf58db909ed3598019fa55ca9fdf44aadab80: Status 404 returned error can't find the container with id 0f06b7a72703424c818563040dadf58db909ed3598019fa55ca9fdf44aadab80 Jul 30 15:51:48.313825 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.313777 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:51:48.313957 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.313861 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:51:48.313957 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.313894 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:51:48.949275 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.949241 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" event={"ID":"d214c878-41c3-437e-ab93-117823bcc7fa","Type":"ContainerStarted","Data":"00bcb7acd277b5d05270a3a4a06730350d6e140dc28b67c0476ed40d21cd0f20"} Jul 30 15:51:48.949687 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.949293 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" event={"ID":"d214c878-41c3-437e-ab93-117823bcc7fa","Type":"ContainerStarted","Data":"0f06b7a72703424c818563040dadf58db909ed3598019fa55ca9fdf44aadab80"} Jul 30 15:51:48.984829 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:48.984766 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" podStartSLOduration=1.9847526009999998 podStartE2EDuration="1.984752601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:51:48.983736522 +0000 UTC m=+2536.218867367" watchObservedRunningTime="2026-07-30 15:51:48.984752601 +0000 UTC m=+2536.219883446" Jul 30 15:51:49.169969 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:49.169936 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:49.175228 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:49.175202 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:49.955049 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:49.955019 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:49.955886 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:49.955863 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:51:52.572423 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:52.572386 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:51:52.983255 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:52.983220 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2"] Jul 30 15:51:52.987247 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:52.987228 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2" Jul 30 15:51:52.991841 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:52.991820 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-db19ef93\"" Jul 30 15:51:53.005029 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.005003 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2"] Jul 30 15:51:53.046107 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.046086 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9"] Jul 30 15:51:53.049520 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.049502 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9" Jul 30 15:51:53.053707 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.053686 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-db19ef93-dockercfg-jhsn9\"" Jul 30 15:51:53.063383 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.063360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrvwd\" (UniqueName: \"kubernetes.io/projected/c3355b31-69e3-4c56-91d7-e7bfb31d8ae5-kube-api-access-xrvwd\") pod \"payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2\" (UID: \"c3355b31-69e3-4c56-91d7-e7bfb31d8ae5\") " pod="openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2" Jul 30 15:51:53.063475 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.063410 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3355b31-69e3-4c56-91d7-e7bfb31d8ae5-plugins-config-volume\") pod \"payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2\" (UID: \"c3355b31-69e3-4c56-91d7-e7bfb31d8ae5\") " pod="openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2" Jul 30 15:51:53.064093 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.064071 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9"] Jul 30 15:51:53.164478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.164444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xtg\" (UniqueName: \"kubernetes.io/projected/6ccf15b8-3c28-42dc-8e84-61444b2e2458-kube-api-access-b9xtg\") pod \"payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9\" (UID: \"6ccf15b8-3c28-42dc-8e84-61444b2e2458\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9" Jul 30 15:51:53.164639 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.164484 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ccf15b8-3c28-42dc-8e84-61444b2e2458-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9\" (UID: \"6ccf15b8-3c28-42dc-8e84-61444b2e2458\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9" Jul 30 15:51:53.164639 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.164510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrvwd\" (UniqueName: \"kubernetes.io/projected/c3355b31-69e3-4c56-91d7-e7bfb31d8ae5-kube-api-access-xrvwd\") pod \"payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2\" (UID: \"c3355b31-69e3-4c56-91d7-e7bfb31d8ae5\") " pod="openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2" Jul 30 15:51:53.164639 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.164600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3355b31-69e3-4c56-91d7-e7bfb31d8ae5-plugins-config-volume\") pod \"payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2\" (UID: \"c3355b31-69e3-4c56-91d7-e7bfb31d8ae5\") " pod="openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2" Jul 30 15:51:53.165271 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.165252 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3355b31-69e3-4c56-91d7-e7bfb31d8ae5-plugins-config-volume\") pod \"payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2\" (UID: \"c3355b31-69e3-4c56-91d7-e7bfb31d8ae5\") " pod="openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2" Jul 30 15:51:53.176157 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.176137 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrvwd\" (UniqueName: \"kubernetes.io/projected/c3355b31-69e3-4c56-91d7-e7bfb31d8ae5-kube-api-access-xrvwd\") pod \"payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2\" (UID: \"c3355b31-69e3-4c56-91d7-e7bfb31d8ae5\") " pod="openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2" Jul 30 15:51:53.265893 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.265821 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ccf15b8-3c28-42dc-8e84-61444b2e2458-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9\" (UID: \"6ccf15b8-3c28-42dc-8e84-61444b2e2458\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9" Jul 30 15:51:53.266017 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.266002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xtg\" (UniqueName: \"kubernetes.io/projected/6ccf15b8-3c28-42dc-8e84-61444b2e2458-kube-api-access-b9xtg\") pod \"payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9\" (UID: \"6ccf15b8-3c28-42dc-8e84-61444b2e2458\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9" Jul 30 15:51:53.266559 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.266541 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ccf15b8-3c28-42dc-8e84-61444b2e2458-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9\" (UID: \"6ccf15b8-3c28-42dc-8e84-61444b2e2458\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9" Jul 30 15:51:53.279742 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.279722 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xtg\" (UniqueName: \"kubernetes.io/projected/6ccf15b8-3c28-42dc-8e84-61444b2e2458-kube-api-access-b9xtg\") pod \"payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9\" (UID: \"6ccf15b8-3c28-42dc-8e84-61444b2e2458\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9" Jul 30 15:51:53.297620 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.297602 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2" Jul 30 15:51:53.359238 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.359180 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9" Jul 30 15:51:53.463296 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.462245 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2"] Jul 30 15:51:53.464445 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:51:53.464405 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3355b31_69e3_4c56_91d7_e7bfb31d8ae5.slice/crio-a23948e624f0f496677ab3b87835e701e72cc80077549f3a6cb384023ed62f85 WatchSource:0}: Error finding container a23948e624f0f496677ab3b87835e701e72cc80077549f3a6cb384023ed62f85: Status 404 returned error can't find the container with id a23948e624f0f496677ab3b87835e701e72cc80077549f3a6cb384023ed62f85 Jul 30 15:51:53.550411 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.550385 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9"] Jul 30 15:51:53.550899 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:51:53.550877 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ccf15b8_3c28_42dc_8e84_61444b2e2458.slice/crio-caea1580bd575ef1d0f1cbdce8801bc0d496a29b37529927f680c5eb93bd6553 WatchSource:0}: Error finding container caea1580bd575ef1d0f1cbdce8801bc0d496a29b37529927f680c5eb93bd6553: Status 404 returned error can't find the container with id caea1580bd575ef1d0f1cbdce8801bc0d496a29b37529927f680c5eb93bd6553 Jul 30 15:51:53.975599 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.975565 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9" event={"ID":"6ccf15b8-3c28-42dc-8e84-61444b2e2458","Type":"ContainerStarted","Data":"cb77c2cd330317829ed238e8f7552b1fa5283a88fcb4efff137e2733d508c4ba"} Jul 30 15:51:53.976101 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.975615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9" event={"ID":"6ccf15b8-3c28-42dc-8e84-61444b2e2458","Type":"ContainerStarted","Data":"caea1580bd575ef1d0f1cbdce8801bc0d496a29b37529927f680c5eb93bd6553"} Jul 30 15:51:53.976101 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.975682 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9" Jul 30 15:51:53.977166 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.977140 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2" event={"ID":"c3355b31-69e3-4c56-91d7-e7bfb31d8ae5","Type":"ContainerStarted","Data":"bc0b4c9e25c9b2094c2f0d9edb15d95dd8bae653bf3c633225c1f010cb0118c2"} Jul 30 15:51:53.977319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.977185 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2" Jul 30 15:51:53.977319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:53.977202 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2" event={"ID":"c3355b31-69e3-4c56-91d7-e7bfb31d8ae5","Type":"ContainerStarted","Data":"a23948e624f0f496677ab3b87835e701e72cc80077549f3a6cb384023ed62f85"} Jul 30 15:51:54.001790 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:51:54.001744 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9" podStartSLOduration=1.001730245 podStartE2EDuration="1.001730245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:51:54.001029617 +0000 UTC m=+2541.236160464" watchObservedRunningTime="2026-07-30 15:51:54.001730245 +0000 UTC m=+2541.236861091" Jul 30 15:52:03.454471 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:03.454398 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2" podStartSLOduration=11.454374192 podStartE2EDuration="11.454374192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:51:54.023251435 +0000 UTC m=+2541.258382280" watchObservedRunningTime="2026-07-30 15:52:03.454374192 +0000 UTC m=+2550.689505131" Jul 30 15:52:03.456703 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:03.456666 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:52:04.984434 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:04.984404 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2" Jul 30 15:52:04.984910 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:04.984584 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9" Jul 30 15:52:09.061111 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:09.061077 2579 generic.go:356] "Generic (PLEG): container finished" podID="d214c878-41c3-437e-ab93-117823bcc7fa" containerID="00bcb7acd277b5d05270a3a4a06730350d6e140dc28b67c0476ed40d21cd0f20" exitCode=0 Jul 30 15:52:09.061660 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:09.061151 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" event={"ID":"d214c878-41c3-437e-ab93-117823bcc7fa","Type":"ContainerDied","Data":"00bcb7acd277b5d05270a3a4a06730350d6e140dc28b67c0476ed40d21cd0f20"} Jul 30 15:52:09.061732 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:09.061666 2579 scope.go:117] "RemoveContainer" containerID="00bcb7acd277b5d05270a3a4a06730350d6e140dc28b67c0476ed40d21cd0f20" Jul 30 15:52:09.062673 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:09.062644 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:52:09.062774 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:09.062714 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:52:09.062774 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:09.062760 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:52:10.072082 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:10.072034 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" event={"ID":"d214c878-41c3-437e-ab93-117823bcc7fa","Type":"ContainerStarted","Data":"a6f32ed652e5868b34f7daa28a5b4d607cdbfbb410a3ebeeb58d670f01c01a01"} Jul 30 15:52:10.170145 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:10.170102 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:52:10.170618 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:10.170576 2579 patch_prober.go:28] interesting pod/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 30 15:52:10.171086 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:10.171056 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" podUID="d214c878-41c3-437e-ab93-117823bcc7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 30 15:52:11.170113 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:11.170078 2579 patch_prober.go:28] interesting pod/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 30 15:52:11.170671 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:11.170639 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" podUID="d214c878-41c3-437e-ab93-117823bcc7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 30 15:52:12.169956 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:12.169912 2579 patch_prober.go:28] interesting pod/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 30 15:52:12.170156 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:12.169988 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" podUID="d214c878-41c3-437e-ab93-117823bcc7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 30 15:52:13.170047 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:13.170012 2579 patch_prober.go:28] interesting pod/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 30 15:52:13.170314 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:13.170286 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" podUID="d214c878-41c3-437e-ab93-117823bcc7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 30 15:52:14.173812 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:14.173766 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:52:14.174272 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:14.174249 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:52:14.174928 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:14.174903 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:52:28.355181 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:28.355150 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:52:46.448428 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:52:46.448389 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:53:20.455597 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:20.455508 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:53:22.205039 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.205004 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2"] Jul 30 15:53:22.205462 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.205186 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2" podUID="c3355b31-69e3-4c56-91d7-e7bfb31d8ae5" containerName="payload-processing" containerID="cri-o://bc0b4c9e25c9b2094c2f0d9edb15d95dd8bae653bf3c633225c1f010cb0118c2" gracePeriod=30 Jul 30 15:53:22.223984 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.223955 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9"] Jul 30 15:53:22.224198 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.224159 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9" podUID="6ccf15b8-3c28-42dc-8e84-61444b2e2458" containerName="payload-pre-processing" containerID="cri-o://cb77c2cd330317829ed238e8f7552b1fa5283a88fcb4efff137e2733d508c4ba" gracePeriod=30 Jul 30 15:53:22.434809 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.434755 2579 generic.go:356] "Generic (PLEG): container finished" podID="6ccf15b8-3c28-42dc-8e84-61444b2e2458" containerID="cb77c2cd330317829ed238e8f7552b1fa5283a88fcb4efff137e2733d508c4ba" exitCode=0 Jul 30 15:53:22.434949 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.434792 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9" event={"ID":"6ccf15b8-3c28-42dc-8e84-61444b2e2458","Type":"ContainerDied","Data":"cb77c2cd330317829ed238e8f7552b1fa5283a88fcb4efff137e2733d508c4ba"} Jul 30 15:53:22.436656 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.436569 2579 generic.go:356] "Generic (PLEG): container finished" podID="c3355b31-69e3-4c56-91d7-e7bfb31d8ae5" containerID="bc0b4c9e25c9b2094c2f0d9edb15d95dd8bae653bf3c633225c1f010cb0118c2" exitCode=0 Jul 30 15:53:22.436656 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.436622 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2" event={"ID":"c3355b31-69e3-4c56-91d7-e7bfb31d8ae5","Type":"ContainerDied","Data":"bc0b4c9e25c9b2094c2f0d9edb15d95dd8bae653bf3c633225c1f010cb0118c2"} Jul 30 15:53:22.476214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.476193 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2" Jul 30 15:53:22.508601 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.508575 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9" Jul 30 15:53:22.563071 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.563041 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ccf15b8-3c28-42dc-8e84-61444b2e2458-plugins-config-volume\") pod \"6ccf15b8-3c28-42dc-8e84-61444b2e2458\" (UID: \"6ccf15b8-3c28-42dc-8e84-61444b2e2458\") " Jul 30 15:53:22.563208 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.563132 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrvwd\" (UniqueName: \"kubernetes.io/projected/c3355b31-69e3-4c56-91d7-e7bfb31d8ae5-kube-api-access-xrvwd\") pod \"c3355b31-69e3-4c56-91d7-e7bfb31d8ae5\" (UID: \"c3355b31-69e3-4c56-91d7-e7bfb31d8ae5\") " Jul 30 15:53:22.563208 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.563185 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9xtg\" (UniqueName: \"kubernetes.io/projected/6ccf15b8-3c28-42dc-8e84-61444b2e2458-kube-api-access-b9xtg\") pod \"6ccf15b8-3c28-42dc-8e84-61444b2e2458\" (UID: \"6ccf15b8-3c28-42dc-8e84-61444b2e2458\") " Jul 30 15:53:22.563326 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.563271 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3355b31-69e3-4c56-91d7-e7bfb31d8ae5-plugins-config-volume\") pod \"c3355b31-69e3-4c56-91d7-e7bfb31d8ae5\" (UID: \"c3355b31-69e3-4c56-91d7-e7bfb31d8ae5\") " Jul 30 15:53:22.563468 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.563438 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ccf15b8-3c28-42dc-8e84-61444b2e2458-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ccf15b8-3c28-42dc-8e84-61444b2e2458" (UID: "6ccf15b8-3c28-42dc-8e84-61444b2e2458"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:53:22.563613 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.563594 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ccf15b8-3c28-42dc-8e84-61444b2e2458-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:53:22.563673 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.563656 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3355b31-69e3-4c56-91d7-e7bfb31d8ae5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3355b31-69e3-4c56-91d7-e7bfb31d8ae5" (UID: "c3355b31-69e3-4c56-91d7-e7bfb31d8ae5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:53:22.565178 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.565158 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3355b31-69e3-4c56-91d7-e7bfb31d8ae5-kube-api-access-xrvwd" (OuterVolumeSpecName: "kube-api-access-xrvwd") pod "c3355b31-69e3-4c56-91d7-e7bfb31d8ae5" (UID: "c3355b31-69e3-4c56-91d7-e7bfb31d8ae5"). InnerVolumeSpecName "kube-api-access-xrvwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:53:22.565267 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.565193 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ccf15b8-3c28-42dc-8e84-61444b2e2458-kube-api-access-b9xtg" (OuterVolumeSpecName: "kube-api-access-b9xtg") pod "6ccf15b8-3c28-42dc-8e84-61444b2e2458" (UID: "6ccf15b8-3c28-42dc-8e84-61444b2e2458"). InnerVolumeSpecName "kube-api-access-b9xtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:53:22.664561 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.664538 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrvwd\" (UniqueName: \"kubernetes.io/projected/c3355b31-69e3-4c56-91d7-e7bfb31d8ae5-kube-api-access-xrvwd\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:53:22.664561 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.664559 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9xtg\" (UniqueName: \"kubernetes.io/projected/6ccf15b8-3c28-42dc-8e84-61444b2e2458-kube-api-access-b9xtg\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:53:22.664693 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:22.664571 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3355b31-69e3-4c56-91d7-e7bfb31d8ae5-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:53:23.442822 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:23.442767 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9" event={"ID":"6ccf15b8-3c28-42dc-8e84-61444b2e2458","Type":"ContainerDied","Data":"caea1580bd575ef1d0f1cbdce8801bc0d496a29b37529927f680c5eb93bd6553"} Jul 30 15:53:23.443214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:23.442850 2579 scope.go:117] "RemoveContainer" containerID="cb77c2cd330317829ed238e8f7552b1fa5283a88fcb4efff137e2733d508c4ba" Jul 30 15:53:23.443214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:23.442784 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9" Jul 30 15:53:23.444182 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:23.444167 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2" Jul 30 15:53:23.444264 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:23.444217 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2" event={"ID":"c3355b31-69e3-4c56-91d7-e7bfb31d8ae5","Type":"ContainerDied","Data":"a23948e624f0f496677ab3b87835e701e72cc80077549f3a6cb384023ed62f85"} Jul 30 15:53:23.451561 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:23.451546 2579 scope.go:117] "RemoveContainer" containerID="bc0b4c9e25c9b2094c2f0d9edb15d95dd8bae653bf3c633225c1f010cb0118c2" Jul 30 15:53:23.468240 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:23.468219 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2"] Jul 30 15:53:23.475585 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:23.475563 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db19ef93-6d44755ff4-jpkw2"] Jul 30 15:53:23.493237 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:23.493218 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9"] Jul 30 15:53:23.498717 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:23.498695 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-db19ef93-5d9b5dcc6-p5wb9"] Jul 30 15:53:25.198356 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:25.198322 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ccf15b8-3c28-42dc-8e84-61444b2e2458" path="/var/lib/kubelet/pods/6ccf15b8-3c28-42dc-8e84-61444b2e2458/volumes" Jul 30 15:53:25.198726 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:25.198683 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3355b31-69e3-4c56-91d7-e7bfb31d8ae5" path="/var/lib/kubelet/pods/c3355b31-69e3-4c56-91d7-e7bfb31d8ae5/volumes" Jul 30 15:53:30.349301 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:53:30.349260 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:54:14.852473 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:14.852434 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j"] Jul 30 15:54:14.853023 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:14.852688 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" podUID="d214c878-41c3-437e-ab93-117823bcc7fa" containerName="istio-proxy" containerID="cri-o://a6f32ed652e5868b34f7daa28a5b4d607cdbfbb410a3ebeeb58d670f01c01a01" gracePeriod=30 Jul 30 15:54:17.749232 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:17.749197 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:54:20.098703 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.098676 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:54:20.176250 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.176222 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f8xv\" (UniqueName: \"kubernetes.io/projected/d214c878-41c3-437e-ab93-117823bcc7fa-kube-api-access-9f8xv\") pod \"d214c878-41c3-437e-ab93-117823bcc7fa\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " Jul 30 15:54:20.176399 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.176278 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-istio-data\") pod \"d214c878-41c3-437e-ab93-117823bcc7fa\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " Jul 30 15:54:20.176399 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.176320 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d214c878-41c3-437e-ab93-117823bcc7fa-istiod-ca-cert\") pod \"d214c878-41c3-437e-ab93-117823bcc7fa\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " Jul 30 15:54:20.176399 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.176343 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-credential-socket\") pod \"d214c878-41c3-437e-ab93-117823bcc7fa\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " Jul 30 15:54:20.176399 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.176363 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-istio-envoy\") pod \"d214c878-41c3-437e-ab93-117823bcc7fa\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " Jul 30 15:54:20.176399 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.176394 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d214c878-41c3-437e-ab93-117823bcc7fa-istio-token\") pod \"d214c878-41c3-437e-ab93-117823bcc7fa\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " Jul 30 15:54:20.176686 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.176419 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-workload-certs\") pod \"d214c878-41c3-437e-ab93-117823bcc7fa\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " Jul 30 15:54:20.176686 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.176453 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d214c878-41c3-437e-ab93-117823bcc7fa-istio-podinfo\") pod \"d214c878-41c3-437e-ab93-117823bcc7fa\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " Jul 30 15:54:20.176686 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.176496 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-workload-socket\") pod \"d214c878-41c3-437e-ab93-117823bcc7fa\" (UID: \"d214c878-41c3-437e-ab93-117823bcc7fa\") " Jul 30 15:54:20.176686 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.176655 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d214c878-41c3-437e-ab93-117823bcc7fa" (UID: "d214c878-41c3-437e-ab93-117823bcc7fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:54:20.176928 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.176740 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d214c878-41c3-437e-ab93-117823bcc7fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d214c878-41c3-437e-ab93-117823bcc7fa" (UID: "d214c878-41c3-437e-ab93-117823bcc7fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:54:20.176928 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.176902 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d214c878-41c3-437e-ab93-117823bcc7fa" (UID: "d214c878-41c3-437e-ab93-117823bcc7fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:54:20.177031 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.176941 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d214c878-41c3-437e-ab93-117823bcc7fa" (UID: "d214c878-41c3-437e-ab93-117823bcc7fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:54:20.177157 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.177134 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d214c878-41c3-437e-ab93-117823bcc7fa-istiod-ca-cert\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:54:20.177286 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.177158 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-credential-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:54:20.177286 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.177174 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-workload-certs\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:54:20.177286 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.177189 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-workload-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:54:20.177472 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.177417 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "d214c878-41c3-437e-ab93-117823bcc7fa" (UID: "d214c878-41c3-437e-ab93-117823bcc7fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:54:20.178593 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.178561 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d214c878-41c3-437e-ab93-117823bcc7fa" (UID: "d214c878-41c3-437e-ab93-117823bcc7fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:54:20.178697 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.178616 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d214c878-41c3-437e-ab93-117823bcc7fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d214c878-41c3-437e-ab93-117823bcc7fa" (UID: "d214c878-41c3-437e-ab93-117823bcc7fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 15:54:20.178697 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.178623 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d214c878-41c3-437e-ab93-117823bcc7fa-kube-api-access-9f8xv" (OuterVolumeSpecName: "kube-api-access-9f8xv") pod "d214c878-41c3-437e-ab93-117823bcc7fa" (UID: "d214c878-41c3-437e-ab93-117823bcc7fa"). InnerVolumeSpecName "kube-api-access-9f8xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:54:20.178873 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.178783 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d214c878-41c3-437e-ab93-117823bcc7fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "d214c878-41c3-437e-ab93-117823bcc7fa" (UID: "d214c878-41c3-437e-ab93-117823bcc7fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:54:20.277836 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.277782 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f8xv\" (UniqueName: \"kubernetes.io/projected/d214c878-41c3-437e-ab93-117823bcc7fa-kube-api-access-9f8xv\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:54:20.277836 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.277831 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-istio-data\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:54:20.277836 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.277840 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d214c878-41c3-437e-ab93-117823bcc7fa-istio-envoy\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:54:20.277836 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.277848 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d214c878-41c3-437e-ab93-117823bcc7fa-istio-token\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:54:20.278098 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.277856 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d214c878-41c3-437e-ab93-117823bcc7fa-istio-podinfo\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:54:20.716183 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.716145 2579 generic.go:356] "Generic (PLEG): container finished" podID="d214c878-41c3-437e-ab93-117823bcc7fa" containerID="a6f32ed652e5868b34f7daa28a5b4d607cdbfbb410a3ebeeb58d670f01c01a01" exitCode=0 Jul 30 15:54:20.716337 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.716200 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" Jul 30 15:54:20.716337 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.716201 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" event={"ID":"d214c878-41c3-437e-ab93-117823bcc7fa","Type":"ContainerDied","Data":"a6f32ed652e5868b34f7daa28a5b4d607cdbfbb410a3ebeeb58d670f01c01a01"} Jul 30 15:54:20.716337 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.716254 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j" event={"ID":"d214c878-41c3-437e-ab93-117823bcc7fa","Type":"ContainerDied","Data":"0f06b7a72703424c818563040dadf58db909ed3598019fa55ca9fdf44aadab80"} Jul 30 15:54:20.716337 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.716284 2579 scope.go:117] "RemoveContainer" containerID="a6f32ed652e5868b34f7daa28a5b4d607cdbfbb410a3ebeeb58d670f01c01a01" Jul 30 15:54:20.727397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.727378 2579 scope.go:117] "RemoveContainer" containerID="00bcb7acd277b5d05270a3a4a06730350d6e140dc28b67c0476ed40d21cd0f20" Jul 30 15:54:20.735222 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.735202 2579 scope.go:117] "RemoveContainer" containerID="a6f32ed652e5868b34f7daa28a5b4d607cdbfbb410a3ebeeb58d670f01c01a01" Jul 30 15:54:20.735473 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:54:20.735455 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6f32ed652e5868b34f7daa28a5b4d607cdbfbb410a3ebeeb58d670f01c01a01\": container with ID starting with a6f32ed652e5868b34f7daa28a5b4d607cdbfbb410a3ebeeb58d670f01c01a01 not found: ID does not exist" containerID="a6f32ed652e5868b34f7daa28a5b4d607cdbfbb410a3ebeeb58d670f01c01a01" Jul 30 15:54:20.735538 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.735481 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f32ed652e5868b34f7daa28a5b4d607cdbfbb410a3ebeeb58d670f01c01a01"} err="failed to get container status \"a6f32ed652e5868b34f7daa28a5b4d607cdbfbb410a3ebeeb58d670f01c01a01\": rpc error: code = NotFound desc = could not find container \"a6f32ed652e5868b34f7daa28a5b4d607cdbfbb410a3ebeeb58d670f01c01a01\": container with ID starting with a6f32ed652e5868b34f7daa28a5b4d607cdbfbb410a3ebeeb58d670f01c01a01 not found: ID does not exist" Jul 30 15:54:20.735538 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.735499 2579 scope.go:117] "RemoveContainer" containerID="00bcb7acd277b5d05270a3a4a06730350d6e140dc28b67c0476ed40d21cd0f20" Jul 30 15:54:20.735745 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:54:20.735725 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00bcb7acd277b5d05270a3a4a06730350d6e140dc28b67c0476ed40d21cd0f20\": container with ID starting with 00bcb7acd277b5d05270a3a4a06730350d6e140dc28b67c0476ed40d21cd0f20 not found: ID does not exist" containerID="00bcb7acd277b5d05270a3a4a06730350d6e140dc28b67c0476ed40d21cd0f20" Jul 30 15:54:20.735786 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.735751 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00bcb7acd277b5d05270a3a4a06730350d6e140dc28b67c0476ed40d21cd0f20"} err="failed to get container status \"00bcb7acd277b5d05270a3a4a06730350d6e140dc28b67c0476ed40d21cd0f20\": rpc error: code = NotFound desc = could not find container \"00bcb7acd277b5d05270a3a4a06730350d6e140dc28b67c0476ed40d21cd0f20\": container with ID starting with 00bcb7acd277b5d05270a3a4a06730350d6e140dc28b67c0476ed40d21cd0f20 not found: ID does not exist" Jul 30 15:54:20.744896 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.744867 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j"] Jul 30 15:54:20.754823 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:20.754786 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-db19ef93-openshift-default-78d8df7d68-xlp8j"] Jul 30 15:54:21.197751 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:21.197722 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d214c878-41c3-437e-ab93-117823bcc7fa" path="/var/lib/kubelet/pods/d214c878-41c3-437e-ab93-117823bcc7fa/volumes" Jul 30 15:54:41.159746 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:41.159704 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:54:50.763244 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:54:50.763207 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:55:34.959718 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:34.959680 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8"] Jul 30 15:55:34.960196 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:34.960065 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ccf15b8-3c28-42dc-8e84-61444b2e2458" containerName="payload-pre-processing" Jul 30 15:55:34.960196 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:34.960077 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ccf15b8-3c28-42dc-8e84-61444b2e2458" containerName="payload-pre-processing" Jul 30 15:55:34.960196 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:34.960091 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d214c878-41c3-437e-ab93-117823bcc7fa" containerName="istio-proxy" Jul 30 15:55:34.960196 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:34.960097 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d214c878-41c3-437e-ab93-117823bcc7fa" containerName="istio-proxy" Jul 30 15:55:34.960196 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:34.960110 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3355b31-69e3-4c56-91d7-e7bfb31d8ae5" containerName="payload-processing" Jul 30 15:55:34.960196 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:34.960116 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3355b31-69e3-4c56-91d7-e7bfb31d8ae5" containerName="payload-processing" Jul 30 15:55:34.960196 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:34.960175 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ccf15b8-3c28-42dc-8e84-61444b2e2458" containerName="payload-pre-processing" Jul 30 15:55:34.960196 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:34.960183 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d214c878-41c3-437e-ab93-117823bcc7fa" containerName="istio-proxy" Jul 30 15:55:34.960196 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:34.960194 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d214c878-41c3-437e-ab93-117823bcc7fa" containerName="istio-proxy" Jul 30 15:55:34.960196 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:34.960201 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3355b31-69e3-4c56-91d7-e7bfb31d8ae5" containerName="payload-processing" Jul 30 15:55:34.963235 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:34.963218 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:34.966682 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:34.966655 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9ef991-openshift-default-dockercfg-jkbz4\"" Jul 30 15:55:34.980368 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:34.980340 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8"] Jul 30 15:55:35.054489 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.054447 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-istio-data\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.054666 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.054497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa17659d-7967-4df4-9b07-86f8faef3189-istio-token\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.054666 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.054578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa17659d-7967-4df4-9b07-86f8faef3189-istio-podinfo\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.054666 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.054617 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-workload-socket\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.054823 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.054718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-credential-socket\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.054823 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.054773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa17659d-7967-4df4-9b07-86f8faef3189-istiod-ca-cert\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.054919 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.054824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqlqv\" (UniqueName: \"kubernetes.io/projected/aa17659d-7967-4df4-9b07-86f8faef3189-kube-api-access-xqlqv\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.054919 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.054900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-workload-certs\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.055005 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.054932 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-istio-envoy\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.155848 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.155820 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-workload-certs\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.156028 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.155857 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-istio-envoy\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.156028 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.155896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-istio-data\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.156028 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.155917 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa17659d-7967-4df4-9b07-86f8faef3189-istio-token\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.156028 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.155951 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa17659d-7967-4df4-9b07-86f8faef3189-istio-podinfo\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.156028 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.155972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-workload-socket\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.156303 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.156037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-credential-socket\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.156303 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.156082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa17659d-7967-4df4-9b07-86f8faef3189-istiod-ca-cert\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.156303 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.156105 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqlqv\" (UniqueName: \"kubernetes.io/projected/aa17659d-7967-4df4-9b07-86f8faef3189-kube-api-access-xqlqv\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.156465 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.156298 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-workload-certs\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.156630 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.156601 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-workload-socket\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.156630 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.156619 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-istio-data\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.156844 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.156675 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-credential-socket\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.156844 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.156818 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa17659d-7967-4df4-9b07-86f8faef3189-istiod-ca-cert\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.158153 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.158129 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-istio-envoy\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.158420 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.158404 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa17659d-7967-4df4-9b07-86f8faef3189-istio-podinfo\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.164864 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.164840 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa17659d-7967-4df4-9b07-86f8faef3189-istio-token\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.166557 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.166536 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqlqv\" (UniqueName: \"kubernetes.io/projected/aa17659d-7967-4df4-9b07-86f8faef3189-kube-api-access-xqlqv\") pod \"e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.272889 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.272778 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:35.410574 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.410538 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8"] Jul 30 15:55:35.411528 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:55:35.411495 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa17659d_7967_4df4_9b07_86f8faef3189.slice/crio-37f190f787388c13af05e151c52a92cb03728afe58a972cf84fb17d8ebfc2159 WatchSource:0}: Error finding container 37f190f787388c13af05e151c52a92cb03728afe58a972cf84fb17d8ebfc2159: Status 404 returned error can't find the container with id 37f190f787388c13af05e151c52a92cb03728afe58a972cf84fb17d8ebfc2159 Jul 30 15:55:35.413688 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.413655 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:55:35.413758 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.413720 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:55:35.413758 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:35.413748 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:55:36.083970 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:36.083936 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" event={"ID":"aa17659d-7967-4df4-9b07-86f8faef3189","Type":"ContainerStarted","Data":"502d33efcd5abd0d26f34f7aa23d50328a1701ed769840163be29fad36933a26"} Jul 30 15:55:36.084365 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:36.083992 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" event={"ID":"aa17659d-7967-4df4-9b07-86f8faef3189","Type":"ContainerStarted","Data":"37f190f787388c13af05e151c52a92cb03728afe58a972cf84fb17d8ebfc2159"} Jul 30 15:55:36.111448 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:36.111397 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" podStartSLOduration=2.111382186 podStartE2EDuration="2.111382186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:55:36.110115802 +0000 UTC m=+2763.345246684" watchObservedRunningTime="2026-07-30 15:55:36.111382186 +0000 UTC m=+2763.346513031" Jul 30 15:55:36.273969 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:36.273932 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:36.278623 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:36.278596 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:37.092764 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:37.092735 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:37.093695 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:37.093674 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:40.138197 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.138159 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m"] Jul 30 15:55:40.141710 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.139192 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d214c878-41c3-437e-ab93-117823bcc7fa" containerName="istio-proxy" Jul 30 15:55:40.141710 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.139312 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d214c878-41c3-437e-ab93-117823bcc7fa" containerName="istio-proxy" Jul 30 15:55:40.144442 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.144416 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m" Jul 30 15:55:40.147313 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.147284 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr"] Jul 30 15:55:40.150715 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.150698 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr" Jul 30 15:55:40.159424 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.159399 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9ef991\"" Jul 30 15:55:40.161214 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.161183 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m"] Jul 30 15:55:40.164090 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.164062 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr"] Jul 30 15:55:40.200650 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.200625 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c03b1e0-e59e-4112-b785-688ad2215ebe-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr\" (UID: \"0c03b1e0-e59e-4112-b785-688ad2215ebe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr" Jul 30 15:55:40.200760 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.200680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swmph\" (UniqueName: \"kubernetes.io/projected/426eb00b-2557-4960-9391-1e06cf85ff0b-kube-api-access-swmph\") pod \"payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m\" (UID: \"426eb00b-2557-4960-9391-1e06cf85ff0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m" Jul 30 15:55:40.200760 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.200737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426eb00b-2557-4960-9391-1e06cf85ff0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m\" (UID: \"426eb00b-2557-4960-9391-1e06cf85ff0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m" Jul 30 15:55:40.200870 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.200774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjg2p\" (UniqueName: \"kubernetes.io/projected/0c03b1e0-e59e-4112-b785-688ad2215ebe-kube-api-access-tjg2p\") pod \"payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr\" (UID: \"0c03b1e0-e59e-4112-b785-688ad2215ebe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr" Jul 30 15:55:40.302290 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.302256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c03b1e0-e59e-4112-b785-688ad2215ebe-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr\" (UID: \"0c03b1e0-e59e-4112-b785-688ad2215ebe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr" Jul 30 15:55:40.302532 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.302319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swmph\" (UniqueName: \"kubernetes.io/projected/426eb00b-2557-4960-9391-1e06cf85ff0b-kube-api-access-swmph\") pod \"payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m\" (UID: \"426eb00b-2557-4960-9391-1e06cf85ff0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m" Jul 30 15:55:40.302532 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.302377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426eb00b-2557-4960-9391-1e06cf85ff0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m\" (UID: \"426eb00b-2557-4960-9391-1e06cf85ff0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m" Jul 30 15:55:40.302532 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.302432 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjg2p\" (UniqueName: \"kubernetes.io/projected/0c03b1e0-e59e-4112-b785-688ad2215ebe-kube-api-access-tjg2p\") pod \"payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr\" (UID: \"0c03b1e0-e59e-4112-b785-688ad2215ebe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr" Jul 30 15:55:40.303124 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.303099 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c03b1e0-e59e-4112-b785-688ad2215ebe-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr\" (UID: \"0c03b1e0-e59e-4112-b785-688ad2215ebe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr" Jul 30 15:55:40.303226 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.303157 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426eb00b-2557-4960-9391-1e06cf85ff0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m\" (UID: \"426eb00b-2557-4960-9391-1e06cf85ff0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m" Jul 30 15:55:40.320840 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.320788 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swmph\" (UniqueName: \"kubernetes.io/projected/426eb00b-2557-4960-9391-1e06cf85ff0b-kube-api-access-swmph\") pod \"payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m\" (UID: \"426eb00b-2557-4960-9391-1e06cf85ff0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m" Jul 30 15:55:40.320941 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.320896 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjg2p\" (UniqueName: \"kubernetes.io/projected/0c03b1e0-e59e-4112-b785-688ad2215ebe-kube-api-access-tjg2p\") pod \"payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr\" (UID: \"0c03b1e0-e59e-4112-b785-688ad2215ebe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr" Jul 30 15:55:40.459015 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.458947 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m" Jul 30 15:55:40.466834 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.466817 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr" Jul 30 15:55:40.611479 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.611452 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m"] Jul 30 15:55:40.613761 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:55:40.613734 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426eb00b_2557_4960_9391_1e06cf85ff0b.slice/crio-bdfb33302c64fefbe79d4ba85dd2ee8e193cee9b8e53f81f0bae20520e47c7ba WatchSource:0}: Error finding container bdfb33302c64fefbe79d4ba85dd2ee8e193cee9b8e53f81f0bae20520e47c7ba: Status 404 returned error can't find the container with id bdfb33302c64fefbe79d4ba85dd2ee8e193cee9b8e53f81f0bae20520e47c7ba Jul 30 15:55:40.633544 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:40.633519 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr"] Jul 30 15:55:40.635323 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:55:40.635300 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c03b1e0_e59e_4112_b785_688ad2215ebe.slice/crio-ecce29762b695004450babf704e2968d70eaa9a446e9873d3e090253cbfb8065 WatchSource:0}: Error finding container ecce29762b695004450babf704e2968d70eaa9a446e9873d3e090253cbfb8065: Status 404 returned error can't find the container with id ecce29762b695004450babf704e2968d70eaa9a446e9873d3e090253cbfb8065 Jul 30 15:55:41.112170 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:41.112130 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m" event={"ID":"426eb00b-2557-4960-9391-1e06cf85ff0b","Type":"ContainerStarted","Data":"56a21fbb8e4177201e8f0ef7d071e6c76a019d659b09488273f30d6e3c68e0df"} Jul 30 15:55:41.112368 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:41.112195 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m" Jul 30 15:55:41.112368 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:41.112219 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m" event={"ID":"426eb00b-2557-4960-9391-1e06cf85ff0b","Type":"ContainerStarted","Data":"bdfb33302c64fefbe79d4ba85dd2ee8e193cee9b8e53f81f0bae20520e47c7ba"} Jul 30 15:55:41.113557 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:41.113533 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr" event={"ID":"0c03b1e0-e59e-4112-b785-688ad2215ebe","Type":"ContainerStarted","Data":"fd9a336b9c8226a264ec86d4ad11c6404a019c04bf132e8cfa360bd8a8692004"} Jul 30 15:55:41.113684 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:41.113586 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr" Jul 30 15:55:41.113684 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:41.113610 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr" event={"ID":"0c03b1e0-e59e-4112-b785-688ad2215ebe","Type":"ContainerStarted","Data":"ecce29762b695004450babf704e2968d70eaa9a446e9873d3e090253cbfb8065"} Jul 30 15:55:41.136246 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:41.136201 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m" podStartSLOduration=1.1361876180000001 podStartE2EDuration="1.136187618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:55:41.134378113 +0000 UTC m=+2768.369508962" watchObservedRunningTime="2026-07-30 15:55:41.136187618 +0000 UTC m=+2768.371318464" Jul 30 15:55:41.155878 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:41.155831 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr" podStartSLOduration=1.155789076 podStartE2EDuration="1.155789076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:55:41.154745031 +0000 UTC m=+2768.389875879" watchObservedRunningTime="2026-07-30 15:55:41.155789076 +0000 UTC m=+2768.390919963" Jul 30 15:55:44.383154 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:44.383117 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:55:52.120350 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:52.120320 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m" Jul 30 15:55:52.120912 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:52.120404 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr" Jul 30 15:55:52.326350 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:52.325487 2579 patch_prober.go:28] interesting pod/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": read tcp 10.132.0.2:41266->10.132.0.51:15021: read: connection reset by peer" start-of-body= Jul 30 15:55:52.326350 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:52.325545 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" podUID="aa17659d-7967-4df4-9b07-86f8faef3189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": read tcp 10.132.0.2:41266->10.132.0.51:15021: read: connection reset by peer" Jul 30 15:55:53.172160 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:53.172130 2579 generic.go:356] "Generic (PLEG): container finished" podID="aa17659d-7967-4df4-9b07-86f8faef3189" containerID="502d33efcd5abd0d26f34f7aa23d50328a1701ed769840163be29fad36933a26" exitCode=0 Jul 30 15:55:53.172556 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:53.172192 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" event={"ID":"aa17659d-7967-4df4-9b07-86f8faef3189","Type":"ContainerDied","Data":"502d33efcd5abd0d26f34f7aa23d50328a1701ed769840163be29fad36933a26"} Jul 30 15:55:53.172556 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:53.172530 2579 scope.go:117] "RemoveContainer" containerID="502d33efcd5abd0d26f34f7aa23d50328a1701ed769840163be29fad36933a26" Jul 30 15:55:53.173553 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:53.173518 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:55:53.173661 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:53.173601 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:55:53.173661 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:53.173652 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:55:54.179594 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:54.179556 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" event={"ID":"aa17659d-7967-4df4-9b07-86f8faef3189","Type":"ContainerStarted","Data":"bb0946a0f54f19fae8807f91b557f8024d11cb546dce54c7c148a5ea150c5385"} Jul 30 15:55:54.273599 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:54.273555 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:54.273832 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:54.273700 2579 patch_prober.go:28] interesting pod/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 30 15:55:54.273832 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:54.273765 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" podUID="aa17659d-7967-4df4-9b07-86f8faef3189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 30 15:55:55.273532 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:55.273488 2579 patch_prober.go:28] interesting pod/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 30 15:55:55.274062 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:55.273552 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" podUID="aa17659d-7967-4df4-9b07-86f8faef3189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 30 15:55:56.273603 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:56.273564 2579 patch_prober.go:28] interesting pod/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 30 15:55:56.274206 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:56.274173 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" podUID="aa17659d-7967-4df4-9b07-86f8faef3189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 30 15:55:57.273199 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:57.273154 2579 patch_prober.go:28] interesting pod/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 30 15:55:57.273420 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:57.273223 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" podUID="aa17659d-7967-4df4-9b07-86f8faef3189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 30 15:55:58.273717 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:58.273679 2579 patch_prober.go:28] interesting pod/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 30 15:55:58.274369 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:58.274333 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" podUID="aa17659d-7967-4df4-9b07-86f8faef3189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 30 15:55:59.276930 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:59.276900 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:59.277312 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:59.277156 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:55:59.277863 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:55:59.277845 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:56:04.000287 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.000258 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7"] Jul 30 15:56:04.003071 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.003053 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.008272 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.008246 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-762215-openshift-default-dockercfg-bkmlp\"" Jul 30 15:56:04.026232 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.026198 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7"] Jul 30 15:56:04.047747 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.047721 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-envoy\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.047886 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.047750 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-workload-socket\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.047886 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.047774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-data\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.047886 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.047873 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-podinfo\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.048036 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.047925 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-workload-certs\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.048036 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.047946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d5acde-a9e5-4adf-b922-6482f58f7607-istiod-ca-cert\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.048036 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.048006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-credential-socket\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.048159 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.048106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-token\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.048159 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.048151 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvxwf\" (UniqueName: \"kubernetes.io/projected/20d5acde-a9e5-4adf-b922-6482f58f7607-kube-api-access-zvxwf\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.148700 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.148663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-credential-socket\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.148922 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.148720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-token\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.148922 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.148742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvxwf\" (UniqueName: \"kubernetes.io/projected/20d5acde-a9e5-4adf-b922-6482f58f7607-kube-api-access-zvxwf\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.148922 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.148769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-envoy\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.148922 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.148790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-workload-socket\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.148922 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.148833 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-data\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.149211 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.149059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-podinfo\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.149211 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.149148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-workload-certs\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.149211 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.149185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d5acde-a9e5-4adf-b922-6482f58f7607-istiod-ca-cert\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.149211 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.149202 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-data\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.149435 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.149146 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-credential-socket\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.149435 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.149261 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-workload-socket\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.149435 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.149396 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-workload-certs\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.149827 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.149783 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d5acde-a9e5-4adf-b922-6482f58f7607-istiod-ca-cert\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.151243 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.151221 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-envoy\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.151464 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.151441 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-podinfo\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.158308 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.158288 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-token\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.158618 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.158597 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvxwf\" (UniqueName: \"kubernetes.io/projected/20d5acde-a9e5-4adf-b922-6482f58f7607-kube-api-access-zvxwf\") pod \"e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.314864 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.314757 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:04.457177 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.457143 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7"] Jul 30 15:56:04.460830 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:56:04.460780 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d5acde_a9e5_4adf_b922_6482f58f7607.slice/crio-1922d4a17cc8a3dafc1c47299590a2dce3ae9e6ed41b2d57583224e4d7cbd7d2 WatchSource:0}: Error finding container 1922d4a17cc8a3dafc1c47299590a2dce3ae9e6ed41b2d57583224e4d7cbd7d2: Status 404 returned error can't find the container with id 1922d4a17cc8a3dafc1c47299590a2dce3ae9e6ed41b2d57583224e4d7cbd7d2 Jul 30 15:56:04.463167 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.463134 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:56:04.463251 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.463218 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:56:04.463251 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:04.463244 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:56:05.232818 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:05.232770 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" event={"ID":"20d5acde-a9e5-4adf-b922-6482f58f7607","Type":"ContainerStarted","Data":"86ac7af1c7f91e0fe2b2d1c5fd3026868bbd626f2be9c969ce1d34727e497076"} Jul 30 15:56:05.233219 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:05.232831 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" event={"ID":"20d5acde-a9e5-4adf-b922-6482f58f7607","Type":"ContainerStarted","Data":"1922d4a17cc8a3dafc1c47299590a2dce3ae9e6ed41b2d57583224e4d7cbd7d2"} Jul 30 15:56:05.261293 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:05.261246 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" podStartSLOduration=2.261229238 podStartE2EDuration="2.261229238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:56:05.260202789 +0000 UTC m=+2792.495333660" watchObservedRunningTime="2026-07-30 15:56:05.261229238 +0000 UTC m=+2792.496360305" Jul 30 15:56:05.315338 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:05.315311 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:05.319946 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:05.319923 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:06.238134 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:06.238096 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:06.239092 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:06.239073 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:09.311877 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:09.311838 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml"] Jul 30 15:56:09.317035 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:09.317012 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml" Jul 30 15:56:09.320465 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:09.320441 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-762215-dockercfg-8f58b\"" Jul 30 15:56:09.320594 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:09.320461 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-762215\"" Jul 30 15:56:09.337464 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:09.337442 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml"] Jul 30 15:56:09.407060 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:09.407030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gltks\" (UniqueName: \"kubernetes.io/projected/5b444b52-332c-42bf-9f11-9792f1e915a9-kube-api-access-gltks\") pod \"payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml\" (UID: \"5b444b52-332c-42bf-9f11-9792f1e915a9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml" Jul 30 15:56:09.407203 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:09.407167 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b444b52-332c-42bf-9f11-9792f1e915a9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml\" (UID: \"5b444b52-332c-42bf-9f11-9792f1e915a9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml" Jul 30 15:56:09.508668 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:09.508636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b444b52-332c-42bf-9f11-9792f1e915a9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml\" (UID: \"5b444b52-332c-42bf-9f11-9792f1e915a9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml" Jul 30 15:56:09.508842 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:09.508688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gltks\" (UniqueName: \"kubernetes.io/projected/5b444b52-332c-42bf-9f11-9792f1e915a9-kube-api-access-gltks\") pod \"payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml\" (UID: \"5b444b52-332c-42bf-9f11-9792f1e915a9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml" Jul 30 15:56:09.509347 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:09.509326 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b444b52-332c-42bf-9f11-9792f1e915a9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml\" (UID: \"5b444b52-332c-42bf-9f11-9792f1e915a9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml" Jul 30 15:56:09.522769 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:09.522747 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gltks\" (UniqueName: \"kubernetes.io/projected/5b444b52-332c-42bf-9f11-9792f1e915a9-kube-api-access-gltks\") pod \"payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml\" (UID: \"5b444b52-332c-42bf-9f11-9792f1e915a9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml" Jul 30 15:56:09.627771 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:09.627745 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml" Jul 30 15:56:09.778184 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:09.778148 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml"] Jul 30 15:56:09.778857 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:56:09.778830 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b444b52_332c_42bf_9f11_9792f1e915a9.slice/crio-c8e3443f81063fb67c570153f3534440611d7d33d5d0451fa4b8c7cd3fca16e5 WatchSource:0}: Error finding container c8e3443f81063fb67c570153f3534440611d7d33d5d0451fa4b8c7cd3fca16e5: Status 404 returned error can't find the container with id c8e3443f81063fb67c570153f3534440611d7d33d5d0451fa4b8c7cd3fca16e5 Jul 30 15:56:10.260178 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:10.260143 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml" event={"ID":"5b444b52-332c-42bf-9f11-9792f1e915a9","Type":"ContainerStarted","Data":"1df31f24c67d47960e6682700718fdb8585adb5c6ecdb42ca283bdfca341e5ab"} Jul 30 15:56:10.260371 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:10.260203 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml" Jul 30 15:56:10.260371 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:10.260219 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml" event={"ID":"5b444b52-332c-42bf-9f11-9792f1e915a9","Type":"ContainerStarted","Data":"c8e3443f81063fb67c570153f3534440611d7d33d5d0451fa4b8c7cd3fca16e5"} Jul 30 15:56:10.285040 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:10.284996 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml" podStartSLOduration=1.284981897 podStartE2EDuration="1.284981897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:56:10.281866828 +0000 UTC m=+2797.516997675" watchObservedRunningTime="2026-07-30 15:56:10.284981897 +0000 UTC m=+2797.520112743" Jul 30 15:56:15.447671 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:15.447637 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:56:21.267585 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:21.267553 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml" Jul 30 15:56:22.279928 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:22.279894 2579 patch_prober.go:28] interesting pod/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 30 15:56:22.280356 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:22.279949 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" podUID="20d5acde-a9e5-4adf-b922-6482f58f7607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 30 15:56:24.330850 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:24.330819 2579 generic.go:356] "Generic (PLEG): container finished" podID="20d5acde-a9e5-4adf-b922-6482f58f7607" containerID="86ac7af1c7f91e0fe2b2d1c5fd3026868bbd626f2be9c969ce1d34727e497076" exitCode=0 Jul 30 15:56:24.331257 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:24.330890 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" event={"ID":"20d5acde-a9e5-4adf-b922-6482f58f7607","Type":"ContainerDied","Data":"86ac7af1c7f91e0fe2b2d1c5fd3026868bbd626f2be9c969ce1d34727e497076"} Jul 30 15:56:24.331311 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:24.331266 2579 scope.go:117] "RemoveContainer" containerID="86ac7af1c7f91e0fe2b2d1c5fd3026868bbd626f2be9c969ce1d34727e497076" Jul 30 15:56:24.332394 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:24.332359 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:56:24.332474 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:24.332426 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:56:24.332513 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:24.332471 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:56:25.337518 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:25.337484 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" event={"ID":"20d5acde-a9e5-4adf-b922-6482f58f7607","Type":"ContainerStarted","Data":"c59523f8d65e7bc07f8abceae733bc922a4058dc102c9f2690cc836b7ec0f2a2"} Jul 30 15:56:26.315192 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:26.315147 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:26.315372 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:26.315268 2579 patch_prober.go:28] interesting pod/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 30 15:56:26.315372 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:26.315315 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" podUID="20d5acde-a9e5-4adf-b922-6482f58f7607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 30 15:56:27.315503 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:27.315469 2579 patch_prober.go:28] interesting pod/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 30 15:56:27.315926 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:27.315521 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" podUID="20d5acde-a9e5-4adf-b922-6482f58f7607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 30 15:56:28.315114 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:28.315078 2579 patch_prober.go:28] interesting pod/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 30 15:56:28.315289 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:28.315146 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" podUID="20d5acde-a9e5-4adf-b922-6482f58f7607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 30 15:56:29.319059 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:29.319029 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:29.319478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:29.319380 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:29.319917 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:29.319898 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:56:44.551968 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:44.551917 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:56:56.240626 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:56:56.240589 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:57:29.741849 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:57:29.741741 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:57:44.040161 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:57:44.040124 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:58:28.642953 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:28.642921 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:58:35.303361 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.303325 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr"] Jul 30 15:58:35.303996 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.303598 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr" podUID="0c03b1e0-e59e-4112-b785-688ad2215ebe" containerName="payload-processing" containerID="cri-o://fd9a336b9c8226a264ec86d4ad11c6404a019c04bf132e8cfa360bd8a8692004" gracePeriod=30 Jul 30 15:58:35.313193 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.313168 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m"] Jul 30 15:58:35.313443 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.313397 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m" podUID="426eb00b-2557-4960-9391-1e06cf85ff0b" containerName="payload-pre-processing" containerID="cri-o://56a21fbb8e4177201e8f0ef7d071e6c76a019d659b09488273f30d6e3c68e0df" gracePeriod=30 Jul 30 15:58:35.577900 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.577868 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr" Jul 30 15:58:35.605836 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.605817 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m" Jul 30 15:58:35.627332 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.627307 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c03b1e0-e59e-4112-b785-688ad2215ebe-plugins-config-volume\") pod \"0c03b1e0-e59e-4112-b785-688ad2215ebe\" (UID: \"0c03b1e0-e59e-4112-b785-688ad2215ebe\") " Jul 30 15:58:35.627512 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.627494 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjg2p\" (UniqueName: \"kubernetes.io/projected/0c03b1e0-e59e-4112-b785-688ad2215ebe-kube-api-access-tjg2p\") pod \"0c03b1e0-e59e-4112-b785-688ad2215ebe\" (UID: \"0c03b1e0-e59e-4112-b785-688ad2215ebe\") " Jul 30 15:58:35.627714 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.627686 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c03b1e0-e59e-4112-b785-688ad2215ebe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0c03b1e0-e59e-4112-b785-688ad2215ebe" (UID: "0c03b1e0-e59e-4112-b785-688ad2215ebe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:58:35.627964 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.627941 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c03b1e0-e59e-4112-b785-688ad2215ebe-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:58:35.629924 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.629898 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c03b1e0-e59e-4112-b785-688ad2215ebe-kube-api-access-tjg2p" (OuterVolumeSpecName: "kube-api-access-tjg2p") pod "0c03b1e0-e59e-4112-b785-688ad2215ebe" (UID: "0c03b1e0-e59e-4112-b785-688ad2215ebe"). InnerVolumeSpecName "kube-api-access-tjg2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:58:35.729147 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.729125 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426eb00b-2557-4960-9391-1e06cf85ff0b-plugins-config-volume\") pod \"426eb00b-2557-4960-9391-1e06cf85ff0b\" (UID: \"426eb00b-2557-4960-9391-1e06cf85ff0b\") " Jul 30 15:58:35.729276 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.729173 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swmph\" (UniqueName: \"kubernetes.io/projected/426eb00b-2557-4960-9391-1e06cf85ff0b-kube-api-access-swmph\") pod \"426eb00b-2557-4960-9391-1e06cf85ff0b\" (UID: \"426eb00b-2557-4960-9391-1e06cf85ff0b\") " Jul 30 15:58:35.729450 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.729434 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjg2p\" (UniqueName: \"kubernetes.io/projected/0c03b1e0-e59e-4112-b785-688ad2215ebe-kube-api-access-tjg2p\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:58:35.729502 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.729441 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426eb00b-2557-4960-9391-1e06cf85ff0b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "426eb00b-2557-4960-9391-1e06cf85ff0b" (UID: "426eb00b-2557-4960-9391-1e06cf85ff0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:58:35.731144 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.731125 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426eb00b-2557-4960-9391-1e06cf85ff0b-kube-api-access-swmph" (OuterVolumeSpecName: "kube-api-access-swmph") pod "426eb00b-2557-4960-9391-1e06cf85ff0b" (UID: "426eb00b-2557-4960-9391-1e06cf85ff0b"). InnerVolumeSpecName "kube-api-access-swmph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:58:35.830216 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.830156 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426eb00b-2557-4960-9391-1e06cf85ff0b-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:58:35.830216 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.830181 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swmph\" (UniqueName: \"kubernetes.io/projected/426eb00b-2557-4960-9391-1e06cf85ff0b-kube-api-access-swmph\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:58:35.989231 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.989198 2579 generic.go:356] "Generic (PLEG): container finished" podID="426eb00b-2557-4960-9391-1e06cf85ff0b" containerID="56a21fbb8e4177201e8f0ef7d071e6c76a019d659b09488273f30d6e3c68e0df" exitCode=0 Jul 30 15:58:35.989366 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.989237 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m" Jul 30 15:58:35.989366 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.989285 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m" event={"ID":"426eb00b-2557-4960-9391-1e06cf85ff0b","Type":"ContainerDied","Data":"56a21fbb8e4177201e8f0ef7d071e6c76a019d659b09488273f30d6e3c68e0df"} Jul 30 15:58:35.989366 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.989335 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m" event={"ID":"426eb00b-2557-4960-9391-1e06cf85ff0b","Type":"ContainerDied","Data":"bdfb33302c64fefbe79d4ba85dd2ee8e193cee9b8e53f81f0bae20520e47c7ba"} Jul 30 15:58:35.989525 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.989369 2579 scope.go:117] "RemoveContainer" containerID="56a21fbb8e4177201e8f0ef7d071e6c76a019d659b09488273f30d6e3c68e0df" Jul 30 15:58:35.990619 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.990597 2579 generic.go:356] "Generic (PLEG): container finished" podID="0c03b1e0-e59e-4112-b785-688ad2215ebe" containerID="fd9a336b9c8226a264ec86d4ad11c6404a019c04bf132e8cfa360bd8a8692004" exitCode=0 Jul 30 15:58:35.990715 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.990643 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr" Jul 30 15:58:35.990765 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.990644 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr" event={"ID":"0c03b1e0-e59e-4112-b785-688ad2215ebe","Type":"ContainerDied","Data":"fd9a336b9c8226a264ec86d4ad11c6404a019c04bf132e8cfa360bd8a8692004"} Jul 30 15:58:35.990765 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.990754 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr" event={"ID":"0c03b1e0-e59e-4112-b785-688ad2215ebe","Type":"ContainerDied","Data":"ecce29762b695004450babf704e2968d70eaa9a446e9873d3e090253cbfb8065"} Jul 30 15:58:35.999439 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.999418 2579 scope.go:117] "RemoveContainer" containerID="56a21fbb8e4177201e8f0ef7d071e6c76a019d659b09488273f30d6e3c68e0df" Jul 30 15:58:35.999720 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:58:35.999703 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56a21fbb8e4177201e8f0ef7d071e6c76a019d659b09488273f30d6e3c68e0df\": container with ID starting with 56a21fbb8e4177201e8f0ef7d071e6c76a019d659b09488273f30d6e3c68e0df not found: ID does not exist" containerID="56a21fbb8e4177201e8f0ef7d071e6c76a019d659b09488273f30d6e3c68e0df" Jul 30 15:58:35.999772 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.999728 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56a21fbb8e4177201e8f0ef7d071e6c76a019d659b09488273f30d6e3c68e0df"} err="failed to get container status \"56a21fbb8e4177201e8f0ef7d071e6c76a019d659b09488273f30d6e3c68e0df\": rpc error: code = NotFound desc = could not find container \"56a21fbb8e4177201e8f0ef7d071e6c76a019d659b09488273f30d6e3c68e0df\": container with ID starting with 56a21fbb8e4177201e8f0ef7d071e6c76a019d659b09488273f30d6e3c68e0df not found: ID does not exist" Jul 30 15:58:35.999772 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:35.999747 2579 scope.go:117] "RemoveContainer" containerID="fd9a336b9c8226a264ec86d4ad11c6404a019c04bf132e8cfa360bd8a8692004" Jul 30 15:58:36.007599 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:36.007583 2579 scope.go:117] "RemoveContainer" containerID="fd9a336b9c8226a264ec86d4ad11c6404a019c04bf132e8cfa360bd8a8692004" Jul 30 15:58:36.007845 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:58:36.007827 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9a336b9c8226a264ec86d4ad11c6404a019c04bf132e8cfa360bd8a8692004\": container with ID starting with fd9a336b9c8226a264ec86d4ad11c6404a019c04bf132e8cfa360bd8a8692004 not found: ID does not exist" containerID="fd9a336b9c8226a264ec86d4ad11c6404a019c04bf132e8cfa360bd8a8692004" Jul 30 15:58:36.007909 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:36.007850 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9a336b9c8226a264ec86d4ad11c6404a019c04bf132e8cfa360bd8a8692004"} err="failed to get container status \"fd9a336b9c8226a264ec86d4ad11c6404a019c04bf132e8cfa360bd8a8692004\": rpc error: code = NotFound desc = could not find container \"fd9a336b9c8226a264ec86d4ad11c6404a019c04bf132e8cfa360bd8a8692004\": container with ID starting with fd9a336b9c8226a264ec86d4ad11c6404a019c04bf132e8cfa360bd8a8692004 not found: ID does not exist" Jul 30 15:58:36.013740 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:36.013720 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m"] Jul 30 15:58:36.017541 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:36.017522 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9ef991-7f98f778f6-kmh7m"] Jul 30 15:58:36.029484 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:36.029465 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr"] Jul 30 15:58:36.034065 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:36.034047 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9ef991-694664d7bd-k5vtr"] Jul 30 15:58:37.197810 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:37.197774 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c03b1e0-e59e-4112-b785-688ad2215ebe" path="/var/lib/kubelet/pods/0c03b1e0-e59e-4112-b785-688ad2215ebe/volumes" Jul 30 15:58:37.198206 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:37.198188 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="426eb00b-2557-4960-9391-1e06cf85ff0b" path="/var/lib/kubelet/pods/426eb00b-2557-4960-9391-1e06cf85ff0b/volumes" Jul 30 15:58:41.923676 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:41.923644 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8"] Jul 30 15:58:41.924205 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:41.923869 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" podUID="aa17659d-7967-4df4-9b07-86f8faef3189" containerName="istio-proxy" containerID="cri-o://bb0946a0f54f19fae8807f91b557f8024d11cb546dce54c7c148a5ea150c5385" gracePeriod=30 Jul 30 15:58:44.280235 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:44.280184 2579 patch_prober.go:28] interesting pod/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 15:58:44.280672 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:44.280261 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" podUID="aa17659d-7967-4df4-9b07-86f8faef3189" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 15:58:44.771365 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:44.771325 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:58:47.052085 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.052048 2579 generic.go:356] "Generic (PLEG): container finished" podID="aa17659d-7967-4df4-9b07-86f8faef3189" containerID="bb0946a0f54f19fae8807f91b557f8024d11cb546dce54c7c148a5ea150c5385" exitCode=0 Jul 30 15:58:47.052440 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.052082 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" event={"ID":"aa17659d-7967-4df4-9b07-86f8faef3189","Type":"ContainerDied","Data":"bb0946a0f54f19fae8807f91b557f8024d11cb546dce54c7c148a5ea150c5385"} Jul 30 15:58:47.052440 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.052147 2579 scope.go:117] "RemoveContainer" containerID="502d33efcd5abd0d26f34f7aa23d50328a1701ed769840163be29fad36933a26" Jul 30 15:58:47.183205 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.183181 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:58:47.332435 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.332352 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-workload-certs\") pod \"aa17659d-7967-4df4-9b07-86f8faef3189\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " Jul 30 15:58:47.332435 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.332390 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-workload-socket\") pod \"aa17659d-7967-4df4-9b07-86f8faef3189\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " Jul 30 15:58:47.332435 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.332416 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqlqv\" (UniqueName: \"kubernetes.io/projected/aa17659d-7967-4df4-9b07-86f8faef3189-kube-api-access-xqlqv\") pod \"aa17659d-7967-4df4-9b07-86f8faef3189\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " Jul 30 15:58:47.332713 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.332477 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa17659d-7967-4df4-9b07-86f8faef3189-istiod-ca-cert\") pod \"aa17659d-7967-4df4-9b07-86f8faef3189\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " Jul 30 15:58:47.332713 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.332508 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-istio-envoy\") pod \"aa17659d-7967-4df4-9b07-86f8faef3189\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " Jul 30 15:58:47.332713 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.332583 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-istio-data\") pod \"aa17659d-7967-4df4-9b07-86f8faef3189\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " Jul 30 15:58:47.332713 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.332641 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa17659d-7967-4df4-9b07-86f8faef3189-istio-podinfo\") pod \"aa17659d-7967-4df4-9b07-86f8faef3189\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " Jul 30 15:58:47.332713 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.332663 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aa17659d-7967-4df4-9b07-86f8faef3189" (UID: "aa17659d-7967-4df4-9b07-86f8faef3189"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:58:47.332713 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.332686 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-credential-socket\") pod \"aa17659d-7967-4df4-9b07-86f8faef3189\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " Jul 30 15:58:47.333045 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.332758 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa17659d-7967-4df4-9b07-86f8faef3189-istio-token\") pod \"aa17659d-7967-4df4-9b07-86f8faef3189\" (UID: \"aa17659d-7967-4df4-9b07-86f8faef3189\") " Jul 30 15:58:47.333109 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.332747 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aa17659d-7967-4df4-9b07-86f8faef3189" (UID: "aa17659d-7967-4df4-9b07-86f8faef3189"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:58:47.333109 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.332968 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aa17659d-7967-4df4-9b07-86f8faef3189" (UID: "aa17659d-7967-4df4-9b07-86f8faef3189"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:58:47.333109 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.332992 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa17659d-7967-4df4-9b07-86f8faef3189-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aa17659d-7967-4df4-9b07-86f8faef3189" (UID: "aa17659d-7967-4df4-9b07-86f8faef3189"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:58:47.333272 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.333208 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-workload-certs\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:58:47.333272 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.333229 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-workload-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:58:47.333272 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.333243 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa17659d-7967-4df4-9b07-86f8faef3189-istiod-ca-cert\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:58:47.333272 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.333257 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-credential-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:58:47.333497 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.333461 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-istio-data" (OuterVolumeSpecName: "istio-data") pod "aa17659d-7967-4df4-9b07-86f8faef3189" (UID: "aa17659d-7967-4df4-9b07-86f8faef3189"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:58:47.334859 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.334834 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa17659d-7967-4df4-9b07-86f8faef3189-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aa17659d-7967-4df4-9b07-86f8faef3189" (UID: "aa17659d-7967-4df4-9b07-86f8faef3189"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 15:58:47.334967 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.334883 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa17659d-7967-4df4-9b07-86f8faef3189-istio-token" (OuterVolumeSpecName: "istio-token") pod "aa17659d-7967-4df4-9b07-86f8faef3189" (UID: "aa17659d-7967-4df4-9b07-86f8faef3189"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:58:47.335179 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.335162 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa17659d-7967-4df4-9b07-86f8faef3189-kube-api-access-xqlqv" (OuterVolumeSpecName: "kube-api-access-xqlqv") pod "aa17659d-7967-4df4-9b07-86f8faef3189" (UID: "aa17659d-7967-4df4-9b07-86f8faef3189"). InnerVolumeSpecName "kube-api-access-xqlqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:58:47.335240 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.335227 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aa17659d-7967-4df4-9b07-86f8faef3189" (UID: "aa17659d-7967-4df4-9b07-86f8faef3189"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:58:47.433960 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.433931 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa17659d-7967-4df4-9b07-86f8faef3189-istio-token\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:58:47.433960 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.433956 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqlqv\" (UniqueName: \"kubernetes.io/projected/aa17659d-7967-4df4-9b07-86f8faef3189-kube-api-access-xqlqv\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:58:47.434127 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.433967 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-istio-envoy\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:58:47.434127 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.433980 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa17659d-7967-4df4-9b07-86f8faef3189-istio-data\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:58:47.434127 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:47.433989 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa17659d-7967-4df4-9b07-86f8faef3189-istio-podinfo\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:58:48.057609 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:48.057579 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" Jul 30 15:58:48.058132 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:48.057618 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8" event={"ID":"aa17659d-7967-4df4-9b07-86f8faef3189","Type":"ContainerDied","Data":"37f190f787388c13af05e151c52a92cb03728afe58a972cf84fb17d8ebfc2159"} Jul 30 15:58:48.058132 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:48.057674 2579 scope.go:117] "RemoveContainer" containerID="bb0946a0f54f19fae8807f91b557f8024d11cb546dce54c7c148a5ea150c5385" Jul 30 15:58:48.082645 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:48.082624 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8"] Jul 30 15:58:48.090360 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:48.090341 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9ef991-openshift-default-79859f678d-vbmw8"] Jul 30 15:58:49.197155 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:49.197116 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa17659d-7967-4df4-9b07-86f8faef3189" path="/var/lib/kubelet/pods/aa17659d-7967-4df4-9b07-86f8faef3189/volumes" Jul 30 15:58:52.838898 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:52.838866 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml"] Jul 30 15:58:52.839388 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:52.839113 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml" podUID="5b444b52-332c-42bf-9f11-9792f1e915a9" containerName="payload-processing" containerID="cri-o://1df31f24c67d47960e6682700718fdb8585adb5c6ecdb42ca283bdfca341e5ab" gracePeriod=30 Jul 30 15:58:53.084144 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:53.084109 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b444b52-332c-42bf-9f11-9792f1e915a9" containerID="1df31f24c67d47960e6682700718fdb8585adb5c6ecdb42ca283bdfca341e5ab" exitCode=0 Jul 30 15:58:53.084313 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:53.084181 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml" event={"ID":"5b444b52-332c-42bf-9f11-9792f1e915a9","Type":"ContainerDied","Data":"1df31f24c67d47960e6682700718fdb8585adb5c6ecdb42ca283bdfca341e5ab"} Jul 30 15:58:53.110252 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:53.110230 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml" Jul 30 15:58:53.284292 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:53.284261 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b444b52-332c-42bf-9f11-9792f1e915a9-plugins-config-volume\") pod \"5b444b52-332c-42bf-9f11-9792f1e915a9\" (UID: \"5b444b52-332c-42bf-9f11-9792f1e915a9\") " Jul 30 15:58:53.284445 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:53.284307 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gltks\" (UniqueName: \"kubernetes.io/projected/5b444b52-332c-42bf-9f11-9792f1e915a9-kube-api-access-gltks\") pod \"5b444b52-332c-42bf-9f11-9792f1e915a9\" (UID: \"5b444b52-332c-42bf-9f11-9792f1e915a9\") " Jul 30 15:58:53.284621 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:53.284593 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b444b52-332c-42bf-9f11-9792f1e915a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5b444b52-332c-42bf-9f11-9792f1e915a9" (UID: "5b444b52-332c-42bf-9f11-9792f1e915a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:58:53.286393 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:53.286371 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b444b52-332c-42bf-9f11-9792f1e915a9-kube-api-access-gltks" (OuterVolumeSpecName: "kube-api-access-gltks") pod "5b444b52-332c-42bf-9f11-9792f1e915a9" (UID: "5b444b52-332c-42bf-9f11-9792f1e915a9"). InnerVolumeSpecName "kube-api-access-gltks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:58:53.385855 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:53.385827 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gltks\" (UniqueName: \"kubernetes.io/projected/5b444b52-332c-42bf-9f11-9792f1e915a9-kube-api-access-gltks\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:58:53.385855 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:53.385852 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b444b52-332c-42bf-9f11-9792f1e915a9-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:58:54.091604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:54.091575 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml" Jul 30 15:58:54.092060 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:54.091600 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml" event={"ID":"5b444b52-332c-42bf-9f11-9792f1e915a9","Type":"ContainerDied","Data":"c8e3443f81063fb67c570153f3534440611d7d33d5d0451fa4b8c7cd3fca16e5"} Jul 30 15:58:54.092060 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:54.091654 2579 scope.go:117] "RemoveContainer" containerID="1df31f24c67d47960e6682700718fdb8585adb5c6ecdb42ca283bdfca341e5ab" Jul 30 15:58:54.116482 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:54.116456 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml"] Jul 30 15:58:54.120972 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:54.120946 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-762215-76b85c6f9d-grkml"] Jul 30 15:58:55.196816 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:55.196775 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b444b52-332c-42bf-9f11-9792f1e915a9" path="/var/lib/kubelet/pods/5b444b52-332c-42bf-9f11-9792f1e915a9/volumes" Jul 30 15:58:57.038076 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:57.038034 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:58:59.591325 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:59.591291 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7"] Jul 30 15:58:59.591832 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:58:59.591484 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" podUID="20d5acde-a9e5-4adf-b922-6482f58f7607" containerName="istio-proxy" containerID="cri-o://c59523f8d65e7bc07f8abceae733bc922a4058dc102c9f2690cc836b7ec0f2a2" gracePeriod=30 Jul 30 15:59:00.527196 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.527168 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr"] Jul 30 15:59:00.527558 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.527546 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa17659d-7967-4df4-9b07-86f8faef3189" containerName="istio-proxy" Jul 30 15:59:00.527604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.527560 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa17659d-7967-4df4-9b07-86f8faef3189" containerName="istio-proxy" Jul 30 15:59:00.527604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.527569 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c03b1e0-e59e-4112-b785-688ad2215ebe" containerName="payload-processing" Jul 30 15:59:00.527604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.527575 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c03b1e0-e59e-4112-b785-688ad2215ebe" containerName="payload-processing" Jul 30 15:59:00.527604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.527595 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426eb00b-2557-4960-9391-1e06cf85ff0b" containerName="payload-pre-processing" Jul 30 15:59:00.527604 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.527601 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="426eb00b-2557-4960-9391-1e06cf85ff0b" containerName="payload-pre-processing" Jul 30 15:59:00.527753 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.527618 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b444b52-332c-42bf-9f11-9792f1e915a9" containerName="payload-processing" Jul 30 15:59:00.527753 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.527625 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b444b52-332c-42bf-9f11-9792f1e915a9" containerName="payload-processing" Jul 30 15:59:00.527753 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.527681 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="426eb00b-2557-4960-9391-1e06cf85ff0b" containerName="payload-pre-processing" Jul 30 15:59:00.527753 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.527689 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c03b1e0-e59e-4112-b785-688ad2215ebe" containerName="payload-processing" Jul 30 15:59:00.527753 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.527695 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b444b52-332c-42bf-9f11-9792f1e915a9" containerName="payload-processing" Jul 30 15:59:00.527753 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.527704 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa17659d-7967-4df4-9b07-86f8faef3189" containerName="istio-proxy" Jul 30 15:59:00.527753 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.527710 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa17659d-7967-4df4-9b07-86f8faef3189" containerName="istio-proxy" Jul 30 15:59:00.531971 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.531954 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.534669 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.534644 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-51545f-openshift-default-dockercfg-658rh\"" Jul 30 15:59:00.546596 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.546565 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr"] Jul 30 15:59:00.548515 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.548491 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-credential-socket\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.548619 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.548586 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlvfc\" (UniqueName: \"kubernetes.io/projected/f89b1c19-f279-4d41-bbc8-df322e5c1986-kube-api-access-vlvfc\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.548688 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.548639 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-token\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.548688 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.548669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89b1c19-f279-4d41-bbc8-df322e5c1986-istiod-ca-cert\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.548779 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.548707 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-podinfo\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.548779 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.548742 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-workload-socket\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.548779 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.548768 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-envoy\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.548915 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.548880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-workload-certs\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.548951 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.548911 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-data\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.649783 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.649749 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlvfc\" (UniqueName: \"kubernetes.io/projected/f89b1c19-f279-4d41-bbc8-df322e5c1986-kube-api-access-vlvfc\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.650273 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.649791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-token\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.650273 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.649846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89b1c19-f279-4d41-bbc8-df322e5c1986-istiod-ca-cert\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.650273 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.649873 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-podinfo\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.650273 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.649914 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-workload-socket\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.650273 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.649946 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-envoy\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.650273 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.650044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-workload-certs\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.650273 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.650074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-data\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.650273 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.650115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-credential-socket\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.650721 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.650414 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-workload-certs\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.650721 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.650459 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-data\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.650721 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.650487 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-workload-socket\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.650721 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.650554 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-credential-socket\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.650721 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.650682 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89b1c19-f279-4d41-bbc8-df322e5c1986-istiod-ca-cert\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.652260 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.652234 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-envoy\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.652374 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.652333 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-podinfo\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.664777 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.664754 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-token\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.664941 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.664924 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlvfc\" (UniqueName: \"kubernetes.io/projected/f89b1c19-f279-4d41-bbc8-df322e5c1986-kube-api-access-vlvfc\") pod \"e2e-api-a-51545f-openshift-default-5b99c567c6-828wr\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.842119 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.842057 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:00.984449 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.984427 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr"] Jul 30 15:59:00.986947 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:59:00.986916 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89b1c19_f279_4d41_bbc8_df322e5c1986.slice/crio-75a4c097699f5179cb492b0e99e56838a58c9bb0cf57c926ac6dcbfe306fa6cf WatchSource:0}: Error finding container 75a4c097699f5179cb492b0e99e56838a58c9bb0cf57c926ac6dcbfe306fa6cf: Status 404 returned error can't find the container with id 75a4c097699f5179cb492b0e99e56838a58c9bb0cf57c926ac6dcbfe306fa6cf Jul 30 15:59:00.989602 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.989571 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:59:00.989691 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.989637 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:59:00.989691 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:00.989665 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:59:01.127752 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:01.127714 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" event={"ID":"f89b1c19-f279-4d41-bbc8-df322e5c1986","Type":"ContainerStarted","Data":"45ebcb0227fd0b03979c67eef166948e441f3fbea68468e5604c927fa050bb66"} Jul 30 15:59:01.127930 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:01.127780 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" event={"ID":"f89b1c19-f279-4d41-bbc8-df322e5c1986","Type":"ContainerStarted","Data":"75a4c097699f5179cb492b0e99e56838a58c9bb0cf57c926ac6dcbfe306fa6cf"} Jul 30 15:59:01.151494 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:01.151439 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" podStartSLOduration=1.15142197 podStartE2EDuration="1.15142197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:59:01.150720836 +0000 UTC m=+2968.385851683" watchObservedRunningTime="2026-07-30 15:59:01.15142197 +0000 UTC m=+2968.386552817" Jul 30 15:59:01.842850 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:01.842787 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:02.847463 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:02.847432 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:03.137920 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:03.137888 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:03.138922 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:03.138899 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:04.847180 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.847157 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:59:04.895516 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.895483 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d5acde-a9e5-4adf-b922-6482f58f7607-istiod-ca-cert\") pod \"20d5acde-a9e5-4adf-b922-6482f58f7607\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " Jul 30 15:59:04.895673 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.895558 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-workload-socket\") pod \"20d5acde-a9e5-4adf-b922-6482f58f7607\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " Jul 30 15:59:04.895673 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.895586 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-data\") pod \"20d5acde-a9e5-4adf-b922-6482f58f7607\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " Jul 30 15:59:04.895673 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.895631 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvxwf\" (UniqueName: \"kubernetes.io/projected/20d5acde-a9e5-4adf-b922-6482f58f7607-kube-api-access-zvxwf\") pod \"20d5acde-a9e5-4adf-b922-6482f58f7607\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " Jul 30 15:59:04.895673 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.895658 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-credential-socket\") pod \"20d5acde-a9e5-4adf-b922-6482f58f7607\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " Jul 30 15:59:04.895961 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.895739 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-envoy\") pod \"20d5acde-a9e5-4adf-b922-6482f58f7607\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " Jul 30 15:59:04.895961 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.895810 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-token\") pod \"20d5acde-a9e5-4adf-b922-6482f58f7607\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " Jul 30 15:59:04.895961 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.895844 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-podinfo\") pod \"20d5acde-a9e5-4adf-b922-6482f58f7607\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " Jul 30 15:59:04.895961 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.895874 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-workload-certs\") pod \"20d5acde-a9e5-4adf-b922-6482f58f7607\" (UID: \"20d5acde-a9e5-4adf-b922-6482f58f7607\") " Jul 30 15:59:04.895961 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.895908 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d5acde-a9e5-4adf-b922-6482f58f7607-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "20d5acde-a9e5-4adf-b922-6482f58f7607" (UID: "20d5acde-a9e5-4adf-b922-6482f58f7607"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:59:04.895961 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.895945 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "20d5acde-a9e5-4adf-b922-6482f58f7607" (UID: "20d5acde-a9e5-4adf-b922-6482f58f7607"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:59:04.896565 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.896544 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d5acde-a9e5-4adf-b922-6482f58f7607-istiod-ca-cert\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:59:04.896710 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.896694 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-workload-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:59:04.896834 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.896580 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "20d5acde-a9e5-4adf-b922-6482f58f7607" (UID: "20d5acde-a9e5-4adf-b922-6482f58f7607"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:59:04.896902 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.896664 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-data" (OuterVolumeSpecName: "istio-data") pod "20d5acde-a9e5-4adf-b922-6482f58f7607" (UID: "20d5acde-a9e5-4adf-b922-6482f58f7607"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:59:04.896902 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.896819 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "20d5acde-a9e5-4adf-b922-6482f58f7607" (UID: "20d5acde-a9e5-4adf-b922-6482f58f7607"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:59:04.898851 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.898811 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "20d5acde-a9e5-4adf-b922-6482f58f7607" (UID: "20d5acde-a9e5-4adf-b922-6482f58f7607"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 15:59:04.898956 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.898904 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "20d5acde-a9e5-4adf-b922-6482f58f7607" (UID: "20d5acde-a9e5-4adf-b922-6482f58f7607"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:59:04.899225 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.899205 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-token" (OuterVolumeSpecName: "istio-token") pod "20d5acde-a9e5-4adf-b922-6482f58f7607" (UID: "20d5acde-a9e5-4adf-b922-6482f58f7607"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:59:04.899295 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.899233 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d5acde-a9e5-4adf-b922-6482f58f7607-kube-api-access-zvxwf" (OuterVolumeSpecName: "kube-api-access-zvxwf") pod "20d5acde-a9e5-4adf-b922-6482f58f7607" (UID: "20d5acde-a9e5-4adf-b922-6482f58f7607"). InnerVolumeSpecName "kube-api-access-zvxwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:59:04.998264 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.998234 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-data\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:59:04.998264 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.998263 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvxwf\" (UniqueName: \"kubernetes.io/projected/20d5acde-a9e5-4adf-b922-6482f58f7607-kube-api-access-zvxwf\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:59:04.998264 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.998276 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-credential-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:59:04.998495 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.998285 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-envoy\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:59:04.998495 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.998293 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-token\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:59:04.998495 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.998301 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d5acde-a9e5-4adf-b922-6482f58f7607-istio-podinfo\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:59:04.998495 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:04.998308 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d5acde-a9e5-4adf-b922-6482f58f7607-workload-certs\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 15:59:05.038151 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.038122 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq"] Jul 30 15:59:05.038514 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.038502 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20d5acde-a9e5-4adf-b922-6482f58f7607" containerName="istio-proxy" Jul 30 15:59:05.038557 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.038515 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="20d5acde-a9e5-4adf-b922-6482f58f7607" containerName="istio-proxy" Jul 30 15:59:05.038557 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.038537 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa17659d-7967-4df4-9b07-86f8faef3189" containerName="istio-proxy" Jul 30 15:59:05.038557 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.038543 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa17659d-7967-4df4-9b07-86f8faef3189" containerName="istio-proxy" Jul 30 15:59:05.038648 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.038616 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="20d5acde-a9e5-4adf-b922-6482f58f7607" containerName="istio-proxy" Jul 30 15:59:05.038648 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.038626 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="20d5acde-a9e5-4adf-b922-6482f58f7607" containerName="istio-proxy" Jul 30 15:59:05.041895 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.041878 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" Jul 30 15:59:05.044529 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.044504 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-51545f\"" Jul 30 15:59:05.052474 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.052446 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq"] Jul 30 15:59:05.059834 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.059810 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn"] Jul 30 15:59:05.060191 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.060179 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20d5acde-a9e5-4adf-b922-6482f58f7607" containerName="istio-proxy" Jul 30 15:59:05.060241 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.060192 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="20d5acde-a9e5-4adf-b922-6482f58f7607" containerName="istio-proxy" Jul 30 15:59:05.063209 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.063194 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn" Jul 30 15:59:05.066412 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.066389 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-51545f-dockercfg-ld9d7\"" Jul 30 15:59:05.075317 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.075289 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn"] Jul 30 15:59:05.098857 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.098833 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d17487c-7cae-4ab3-b8d8-733d0bab643e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn\" (UID: \"1d17487c-7cae-4ab3-b8d8-733d0bab643e\") " pod="openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn" Jul 30 15:59:05.098972 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.098868 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76p5x\" (UniqueName: \"kubernetes.io/projected/96010de1-983a-4e58-b517-6ee77b6fc760-kube-api-access-76p5x\") pod \"payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq\" (UID: \"96010de1-983a-4e58-b517-6ee77b6fc760\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" Jul 30 15:59:05.098972 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.098947 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbvbh\" (UniqueName: \"kubernetes.io/projected/1d17487c-7cae-4ab3-b8d8-733d0bab643e-kube-api-access-hbvbh\") pod \"payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn\" (UID: \"1d17487c-7cae-4ab3-b8d8-733d0bab643e\") " pod="openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn" Jul 30 15:59:05.099058 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.099045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96010de1-983a-4e58-b517-6ee77b6fc760-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq\" (UID: \"96010de1-983a-4e58-b517-6ee77b6fc760\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" Jul 30 15:59:05.151613 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.151534 2579 generic.go:356] "Generic (PLEG): container finished" podID="20d5acde-a9e5-4adf-b922-6482f58f7607" containerID="c59523f8d65e7bc07f8abceae733bc922a4058dc102c9f2690cc836b7ec0f2a2" exitCode=0 Jul 30 15:59:05.151613 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.151583 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" Jul 30 15:59:05.151839 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.151618 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" event={"ID":"20d5acde-a9e5-4adf-b922-6482f58f7607","Type":"ContainerDied","Data":"c59523f8d65e7bc07f8abceae733bc922a4058dc102c9f2690cc836b7ec0f2a2"} Jul 30 15:59:05.151839 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.151672 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7" event={"ID":"20d5acde-a9e5-4adf-b922-6482f58f7607","Type":"ContainerDied","Data":"1922d4a17cc8a3dafc1c47299590a2dce3ae9e6ed41b2d57583224e4d7cbd7d2"} Jul 30 15:59:05.151839 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.151696 2579 scope.go:117] "RemoveContainer" containerID="c59523f8d65e7bc07f8abceae733bc922a4058dc102c9f2690cc836b7ec0f2a2" Jul 30 15:59:05.165930 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.165904 2579 scope.go:117] "RemoveContainer" containerID="86ac7af1c7f91e0fe2b2d1c5fd3026868bbd626f2be9c969ce1d34727e497076" Jul 30 15:59:05.177102 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.177009 2579 scope.go:117] "RemoveContainer" containerID="c59523f8d65e7bc07f8abceae733bc922a4058dc102c9f2690cc836b7ec0f2a2" Jul 30 15:59:05.177590 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:59:05.177555 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59523f8d65e7bc07f8abceae733bc922a4058dc102c9f2690cc836b7ec0f2a2\": container with ID starting with c59523f8d65e7bc07f8abceae733bc922a4058dc102c9f2690cc836b7ec0f2a2 not found: ID does not exist" containerID="c59523f8d65e7bc07f8abceae733bc922a4058dc102c9f2690cc836b7ec0f2a2" Jul 30 15:59:05.177686 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.177605 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59523f8d65e7bc07f8abceae733bc922a4058dc102c9f2690cc836b7ec0f2a2"} err="failed to get container status \"c59523f8d65e7bc07f8abceae733bc922a4058dc102c9f2690cc836b7ec0f2a2\": rpc error: code = NotFound desc = could not find container \"c59523f8d65e7bc07f8abceae733bc922a4058dc102c9f2690cc836b7ec0f2a2\": container with ID starting with c59523f8d65e7bc07f8abceae733bc922a4058dc102c9f2690cc836b7ec0f2a2 not found: ID does not exist" Jul 30 15:59:05.177686 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.177633 2579 scope.go:117] "RemoveContainer" containerID="86ac7af1c7f91e0fe2b2d1c5fd3026868bbd626f2be9c969ce1d34727e497076" Jul 30 15:59:05.178086 ip-10-0-142-244 kubenswrapper[2579]: E0730 15:59:05.178038 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ac7af1c7f91e0fe2b2d1c5fd3026868bbd626f2be9c969ce1d34727e497076\": container with ID starting with 86ac7af1c7f91e0fe2b2d1c5fd3026868bbd626f2be9c969ce1d34727e497076 not found: ID does not exist" containerID="86ac7af1c7f91e0fe2b2d1c5fd3026868bbd626f2be9c969ce1d34727e497076" Jul 30 15:59:05.178086 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.178077 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ac7af1c7f91e0fe2b2d1c5fd3026868bbd626f2be9c969ce1d34727e497076"} err="failed to get container status \"86ac7af1c7f91e0fe2b2d1c5fd3026868bbd626f2be9c969ce1d34727e497076\": rpc error: code = NotFound desc = could not find container \"86ac7af1c7f91e0fe2b2d1c5fd3026868bbd626f2be9c969ce1d34727e497076\": container with ID starting with 86ac7af1c7f91e0fe2b2d1c5fd3026868bbd626f2be9c969ce1d34727e497076 not found: ID does not exist" Jul 30 15:59:05.183399 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.183376 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7"] Jul 30 15:59:05.188705 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.188682 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-762215-openshift-default-7fbb88c6db-6b8t7"] Jul 30 15:59:05.198319 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.198296 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20d5acde-a9e5-4adf-b922-6482f58f7607" path="/var/lib/kubelet/pods/20d5acde-a9e5-4adf-b922-6482f58f7607/volumes" Jul 30 15:59:05.200473 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.200453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d17487c-7cae-4ab3-b8d8-733d0bab643e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn\" (UID: \"1d17487c-7cae-4ab3-b8d8-733d0bab643e\") " pod="openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn" Jul 30 15:59:05.200559 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.200501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76p5x\" (UniqueName: \"kubernetes.io/projected/96010de1-983a-4e58-b517-6ee77b6fc760-kube-api-access-76p5x\") pod \"payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq\" (UID: \"96010de1-983a-4e58-b517-6ee77b6fc760\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" Jul 30 15:59:05.200615 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.200558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbvbh\" (UniqueName: \"kubernetes.io/projected/1d17487c-7cae-4ab3-b8d8-733d0bab643e-kube-api-access-hbvbh\") pod \"payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn\" (UID: \"1d17487c-7cae-4ab3-b8d8-733d0bab643e\") " pod="openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn" Jul 30 15:59:05.200674 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.200623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96010de1-983a-4e58-b517-6ee77b6fc760-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq\" (UID: \"96010de1-983a-4e58-b517-6ee77b6fc760\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" Jul 30 15:59:05.201327 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.201308 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d17487c-7cae-4ab3-b8d8-733d0bab643e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn\" (UID: \"1d17487c-7cae-4ab3-b8d8-733d0bab643e\") " pod="openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn" Jul 30 15:59:05.201402 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.201325 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96010de1-983a-4e58-b517-6ee77b6fc760-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq\" (UID: \"96010de1-983a-4e58-b517-6ee77b6fc760\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" Jul 30 15:59:05.209374 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.209341 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76p5x\" (UniqueName: \"kubernetes.io/projected/96010de1-983a-4e58-b517-6ee77b6fc760-kube-api-access-76p5x\") pod \"payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq\" (UID: \"96010de1-983a-4e58-b517-6ee77b6fc760\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" Jul 30 15:59:05.209462 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.209342 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbvbh\" (UniqueName: \"kubernetes.io/projected/1d17487c-7cae-4ab3-b8d8-733d0bab643e-kube-api-access-hbvbh\") pod \"payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn\" (UID: \"1d17487c-7cae-4ab3-b8d8-733d0bab643e\") " pod="openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn" Jul 30 15:59:05.353738 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.353705 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" Jul 30 15:59:05.374355 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.374326 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn" Jul 30 15:59:05.493626 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.493547 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq"] Jul 30 15:59:05.498842 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:59:05.498785 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96010de1_983a_4e58_b517_6ee77b6fc760.slice/crio-7aabdd83c4a62c6a0cd2c80fba2a5e8a4ac65697d06efea5ba03fcdc9c77e77a WatchSource:0}: Error finding container 7aabdd83c4a62c6a0cd2c80fba2a5e8a4ac65697d06efea5ba03fcdc9c77e77a: Status 404 returned error can't find the container with id 7aabdd83c4a62c6a0cd2c80fba2a5e8a4ac65697d06efea5ba03fcdc9c77e77a Jul 30 15:59:05.527764 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:05.527741 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn"] Jul 30 15:59:05.537874 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:59:05.537851 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d17487c_7cae_4ab3_b8d8_733d0bab643e.slice/crio-28697911c1f388b15aae72c61d6d8e1e8217a5407460a07f1ef3e73412c088b6 WatchSource:0}: Error finding container 28697911c1f388b15aae72c61d6d8e1e8217a5407460a07f1ef3e73412c088b6: Status 404 returned error can't find the container with id 28697911c1f388b15aae72c61d6d8e1e8217a5407460a07f1ef3e73412c088b6 Jul 30 15:59:06.160478 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:06.160435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn" event={"ID":"1d17487c-7cae-4ab3-b8d8-733d0bab643e","Type":"ContainerStarted","Data":"52e339017b9f93fd06dc70956da53b68fa59651d7b16234d1d3083f69788d77a"} Jul 30 15:59:06.160954 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:06.160498 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn" event={"ID":"1d17487c-7cae-4ab3-b8d8-733d0bab643e","Type":"ContainerStarted","Data":"28697911c1f388b15aae72c61d6d8e1e8217a5407460a07f1ef3e73412c088b6"} Jul 30 15:59:06.160954 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:06.160532 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn" Jul 30 15:59:06.162081 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:06.162060 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" event={"ID":"96010de1-983a-4e58-b517-6ee77b6fc760","Type":"ContainerStarted","Data":"352281391acbfb5e2e82352a69844a9e845ab57530adf7dc6d32e70f2e486e9f"} Jul 30 15:59:06.162195 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:06.162097 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" event={"ID":"96010de1-983a-4e58-b517-6ee77b6fc760","Type":"ContainerStarted","Data":"7aabdd83c4a62c6a0cd2c80fba2a5e8a4ac65697d06efea5ba03fcdc9c77e77a"} Jul 30 15:59:06.162248 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:06.162195 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" Jul 30 15:59:06.182517 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:06.182473 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn" podStartSLOduration=1.182459369 podStartE2EDuration="1.182459369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:59:06.181183955 +0000 UTC m=+2973.416314802" watchObservedRunningTime="2026-07-30 15:59:06.182459369 +0000 UTC m=+2973.417590215" Jul 30 15:59:06.202049 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:06.202005 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" podStartSLOduration=1.201992469 podStartE2EDuration="1.201992469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:59:06.201333784 +0000 UTC m=+2973.436464631" watchObservedRunningTime="2026-07-30 15:59:06.201992469 +0000 UTC m=+2973.437123315" Jul 30 15:59:07.843128 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:07.843097 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:59:16.213844 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:16.213813 2579 generic.go:356] "Generic (PLEG): container finished" podID="f89b1c19-f279-4d41-bbc8-df322e5c1986" containerID="45ebcb0227fd0b03979c67eef166948e441f3fbea68468e5604c927fa050bb66" exitCode=0 Jul 30 15:59:16.214193 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:16.213833 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" event={"ID":"f89b1c19-f279-4d41-bbc8-df322e5c1986","Type":"ContainerDied","Data":"45ebcb0227fd0b03979c67eef166948e441f3fbea68468e5604c927fa050bb66"} Jul 30 15:59:16.214237 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:16.214219 2579 scope.go:117] "RemoveContainer" containerID="45ebcb0227fd0b03979c67eef166948e441f3fbea68468e5604c927fa050bb66" Jul 30 15:59:16.215211 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:16.215183 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:59:16.215276 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:16.215251 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:59:16.215315 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:16.215280 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:59:17.169247 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:17.169200 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" podUID="96010de1-983a-4e58-b517-6ee77b6fc760" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.57:9004: connect: connection refused" Jul 30 15:59:17.169863 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:17.169839 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn" Jul 30 15:59:17.221699 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:17.221671 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" event={"ID":"f89b1c19-f279-4d41-bbc8-df322e5c1986","Type":"ContainerStarted","Data":"276b66cf5d15c04bfe271827a88ea724a0f59fc614e2b86b35dc547e28191a36"} Jul 30 15:59:17.843222 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:17.843187 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:17.844368 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:17.844339 2579 patch_prober.go:28] interesting pod/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 30 15:59:17.844509 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:17.844408 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" podUID="f89b1c19-f279-4d41-bbc8-df322e5c1986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 30 15:59:18.138698 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:18.138663 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:18.842886 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:18.842853 2579 patch_prober.go:28] interesting pod/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 30 15:59:18.843262 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:18.842902 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" podUID="f89b1c19-f279-4d41-bbc8-df322e5c1986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 30 15:59:19.843160 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:19.843122 2579 patch_prober.go:28] interesting pod/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 30 15:59:19.843614 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:19.843180 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" podUID="f89b1c19-f279-4d41-bbc8-df322e5c1986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 30 15:59:20.843231 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:20.843194 2579 patch_prober.go:28] interesting pod/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 30 15:59:20.843622 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:20.843246 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" podUID="f89b1c19-f279-4d41-bbc8-df322e5c1986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 30 15:59:21.847277 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:21.847242 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:21.848226 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:21.848202 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 15:59:27.169141 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:27.169112 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" Jul 30 15:59:28.972995 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:28.972961 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9"] Jul 30 15:59:28.976423 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:28.976403 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:28.979178 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:28.979157 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-985263-openshift-default-dockercfg-nh7j6\"" Jul 30 15:59:28.988507 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:28.988482 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9"] Jul 30 15:59:29.135427 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.135391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2b6f\" (UniqueName: \"kubernetes.io/projected/1da12581-0837-4ada-ab71-9dfb216cb4fc-kube-api-access-q2b6f\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.135599 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.135444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-podinfo\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.135599 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.135556 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-token\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.135675 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.135605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-envoy\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.135710 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.135668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-workload-certs\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.135750 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.135715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-workload-socket\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.135785 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.135754 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-credential-socket\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.135785 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.135778 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-data\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.135883 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.135828 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1da12581-0837-4ada-ab71-9dfb216cb4fc-istiod-ca-cert\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.237471 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.237394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-envoy\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.237723 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.237705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-workload-certs\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.238144 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.238116 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-workload-socket\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.238372 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.238330 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-workload-certs\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.238482 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.238404 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-workload-socket\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.238482 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.238475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-credential-socket\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.238592 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.238506 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-data\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.238592 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.238541 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1da12581-0837-4ada-ab71-9dfb216cb4fc-istiod-ca-cert\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.238592 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.238586 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2b6f\" (UniqueName: \"kubernetes.io/projected/1da12581-0837-4ada-ab71-9dfb216cb4fc-kube-api-access-q2b6f\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.238713 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.238647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-podinfo\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.238754 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.238724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-token\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.238817 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.238724 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-data\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.239228 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.239204 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-credential-socket\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.239351 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.239331 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1da12581-0837-4ada-ab71-9dfb216cb4fc-istiod-ca-cert\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.240109 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.240082 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-envoy\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.241190 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.241169 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-podinfo\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.247037 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.247012 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-token\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.247123 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.247040 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2b6f\" (UniqueName: \"kubernetes.io/projected/1da12581-0837-4ada-ab71-9dfb216cb4fc-kube-api-access-q2b6f\") pod \"e2e-api-b-985263-openshift-default-6b6ddcd576-955z9\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.286656 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.286635 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:29.418099 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.418076 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9"] Jul 30 15:59:29.419968 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:59:29.419941 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da12581_0837_4ada_ab71_9dfb216cb4fc.slice/crio-833cc89ed1f7a490a4decb00965ddc9f13570af37020c98945c6ecffe093dd19 WatchSource:0}: Error finding container 833cc89ed1f7a490a4decb00965ddc9f13570af37020c98945c6ecffe093dd19: Status 404 returned error can't find the container with id 833cc89ed1f7a490a4decb00965ddc9f13570af37020c98945c6ecffe093dd19 Jul 30 15:59:29.422107 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.422071 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:59:29.422234 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.422140 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:59:29.422234 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:29.422167 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:59:30.290643 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:30.290607 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" event={"ID":"1da12581-0837-4ada-ab71-9dfb216cb4fc","Type":"ContainerStarted","Data":"0e51cb5053d5896d474a52502999f4ba1225600c0aac0c865f5da602f72f4b56"} Jul 30 15:59:30.291060 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:30.290667 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" event={"ID":"1da12581-0837-4ada-ab71-9dfb216cb4fc","Type":"ContainerStarted","Data":"833cc89ed1f7a490a4decb00965ddc9f13570af37020c98945c6ecffe093dd19"} Jul 30 15:59:30.319465 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:30.319419 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" podStartSLOduration=2.319405235 podStartE2EDuration="2.319405235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:59:30.317557534 +0000 UTC m=+2997.552688380" watchObservedRunningTime="2026-07-30 15:59:30.319405235 +0000 UTC m=+2997.554536081" Jul 30 15:59:31.287463 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:31.287428 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:31.292525 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:31.292501 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:31.295716 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:31.295694 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:31.296345 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:31.296326 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:34.385945 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.385906 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt"] Jul 30 15:59:34.391502 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.391476 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt" Jul 30 15:59:34.394954 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.394935 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-985263\"" Jul 30 15:59:34.414442 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.414418 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt"] Jul 30 15:59:34.477633 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.477603 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n28j\" (UniqueName: \"kubernetes.io/projected/d98e5399-d19b-4b38-8cb0-0921222abced-kube-api-access-8n28j\") pod \"payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt\" (UID: \"d98e5399-d19b-4b38-8cb0-0921222abced\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt" Jul 30 15:59:34.477754 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.477650 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98e5399-d19b-4b38-8cb0-0921222abced-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt\" (UID: \"d98e5399-d19b-4b38-8cb0-0921222abced\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt" Jul 30 15:59:34.480879 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.480854 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg"] Jul 30 15:59:34.483695 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.483680 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg" Jul 30 15:59:34.492067 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.492049 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-985263-dockercfg-p7kz8\"" Jul 30 15:59:34.509432 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.509406 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg"] Jul 30 15:59:34.578495 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.578465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n28j\" (UniqueName: \"kubernetes.io/projected/d98e5399-d19b-4b38-8cb0-0921222abced-kube-api-access-8n28j\") pod \"payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt\" (UID: \"d98e5399-d19b-4b38-8cb0-0921222abced\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt" Jul 30 15:59:34.578602 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.578525 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfw64\" (UniqueName: \"kubernetes.io/projected/d88f484e-a906-4390-aa5c-7864115a2d9a-kube-api-access-tfw64\") pod \"payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg\" (UID: \"d88f484e-a906-4390-aa5c-7864115a2d9a\") " pod="openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg" Jul 30 15:59:34.578602 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.578565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98e5399-d19b-4b38-8cb0-0921222abced-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt\" (UID: \"d98e5399-d19b-4b38-8cb0-0921222abced\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt" Jul 30 15:59:34.578696 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.578675 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88f484e-a906-4390-aa5c-7864115a2d9a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg\" (UID: \"d88f484e-a906-4390-aa5c-7864115a2d9a\") " pod="openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg" Jul 30 15:59:34.579235 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.579219 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98e5399-d19b-4b38-8cb0-0921222abced-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt\" (UID: \"d98e5399-d19b-4b38-8cb0-0921222abced\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt" Jul 30 15:59:34.619397 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.619368 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n28j\" (UniqueName: \"kubernetes.io/projected/d98e5399-d19b-4b38-8cb0-0921222abced-kube-api-access-8n28j\") pod \"payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt\" (UID: \"d98e5399-d19b-4b38-8cb0-0921222abced\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt" Jul 30 15:59:34.679217 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.679153 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfw64\" (UniqueName: \"kubernetes.io/projected/d88f484e-a906-4390-aa5c-7864115a2d9a-kube-api-access-tfw64\") pod \"payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg\" (UID: \"d88f484e-a906-4390-aa5c-7864115a2d9a\") " pod="openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg" Jul 30 15:59:34.679314 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.679255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88f484e-a906-4390-aa5c-7864115a2d9a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg\" (UID: \"d88f484e-a906-4390-aa5c-7864115a2d9a\") " pod="openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg" Jul 30 15:59:34.679931 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.679910 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88f484e-a906-4390-aa5c-7864115a2d9a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg\" (UID: \"d88f484e-a906-4390-aa5c-7864115a2d9a\") " pod="openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg" Jul 30 15:59:34.691348 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.691329 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfw64\" (UniqueName: \"kubernetes.io/projected/d88f484e-a906-4390-aa5c-7864115a2d9a-kube-api-access-tfw64\") pod \"payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg\" (UID: \"d88f484e-a906-4390-aa5c-7864115a2d9a\") " pod="openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg" Jul 30 15:59:34.701515 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.701499 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt" Jul 30 15:59:34.794877 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.794840 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg" Jul 30 15:59:34.838790 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.838766 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt"] Jul 30 15:59:34.939388 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:34.939366 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg"] Jul 30 15:59:34.941168 ip-10-0-142-244 kubenswrapper[2579]: W0730 15:59:34.941147 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88f484e_a906_4390_aa5c_7864115a2d9a.slice/crio-26491c06a5939245e2a42ec7c9f2340a49429dd15fb83bfb521f427dcafe42f2 WatchSource:0}: Error finding container 26491c06a5939245e2a42ec7c9f2340a49429dd15fb83bfb521f427dcafe42f2: Status 404 returned error can't find the container with id 26491c06a5939245e2a42ec7c9f2340a49429dd15fb83bfb521f427dcafe42f2 Jul 30 15:59:35.318787 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:35.318698 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg" event={"ID":"d88f484e-a906-4390-aa5c-7864115a2d9a","Type":"ContainerStarted","Data":"5b873bff2fc8cb9e976da979ff993ed0fae84e81a878d1ac67796def0e2ae66c"} Jul 30 15:59:35.318787 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:35.318756 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg" event={"ID":"d88f484e-a906-4390-aa5c-7864115a2d9a","Type":"ContainerStarted","Data":"26491c06a5939245e2a42ec7c9f2340a49429dd15fb83bfb521f427dcafe42f2"} Jul 30 15:59:35.319065 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:35.318815 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg" Jul 30 15:59:35.320226 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:35.320197 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt" event={"ID":"d98e5399-d19b-4b38-8cb0-0921222abced","Type":"ContainerStarted","Data":"46fd10c03228691a14c20844aeaf5ad2ca2ac94a0a686e5ca89daad9e9b8e0e2"} Jul 30 15:59:35.320333 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:35.320244 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt" event={"ID":"d98e5399-d19b-4b38-8cb0-0921222abced","Type":"ContainerStarted","Data":"d8a54066739ff7201903a2d37f4bbd2f4189f31b38e7a1bb7f2b17b1ddeee50b"} Jul 30 15:59:35.320333 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:35.320281 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt" Jul 30 15:59:35.343271 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:35.343227 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg" podStartSLOduration=1.343213845 podStartE2EDuration="1.343213845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:59:35.341450666 +0000 UTC m=+3002.576581512" watchObservedRunningTime="2026-07-30 15:59:35.343213845 +0000 UTC m=+3002.578344692" Jul 30 15:59:35.362351 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:35.362309 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt" podStartSLOduration=1.362296503 podStartE2EDuration="1.362296503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:59:35.361900758 +0000 UTC m=+3002.597031604" watchObservedRunningTime="2026-07-30 15:59:35.362296503 +0000 UTC m=+3002.597427351" Jul 30 15:59:39.941691 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:39.941659 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 15:59:46.326996 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:46.326962 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt" Jul 30 15:59:46.327487 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:46.327411 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg" Jul 30 15:59:47.336978 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:47.336936 2579 patch_prober.go:28] interesting pod/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 30 15:59:47.337504 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:47.337006 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" podUID="1da12581-0837-4ada-ab71-9dfb216cb4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 30 15:59:48.402482 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:48.402452 2579 generic.go:356] "Generic (PLEG): container finished" podID="1da12581-0837-4ada-ab71-9dfb216cb4fc" containerID="0e51cb5053d5896d474a52502999f4ba1225600c0aac0c865f5da602f72f4b56" exitCode=0 Jul 30 15:59:48.402907 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:48.402488 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" event={"ID":"1da12581-0837-4ada-ab71-9dfb216cb4fc","Type":"ContainerDied","Data":"0e51cb5053d5896d474a52502999f4ba1225600c0aac0c865f5da602f72f4b56"} Jul 30 15:59:48.402907 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:48.402865 2579 scope.go:117] "RemoveContainer" containerID="0e51cb5053d5896d474a52502999f4ba1225600c0aac0c865f5da602f72f4b56" Jul 30 15:59:48.404035 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:48.404002 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:59:48.404112 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:48.404087 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:59:48.404152 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:48.404122 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 15:59:49.410579 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:49.410546 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" event={"ID":"1da12581-0837-4ada-ab71-9dfb216cb4fc","Type":"ContainerStarted","Data":"d110946e2c8d6d441691b9eab84aa933b7a5a874505424e16eccd90afba29880"} Jul 30 15:59:50.287909 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:50.287859 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:50.288157 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:50.288131 2579 patch_prober.go:28] interesting pod/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 30 15:59:50.288251 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:50.288191 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" podUID="1da12581-0837-4ada-ab71-9dfb216cb4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 30 15:59:51.287581 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:51.287546 2579 patch_prober.go:28] interesting pod/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 30 15:59:51.287996 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:51.287599 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" podUID="1da12581-0837-4ada-ab71-9dfb216cb4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 30 15:59:52.287784 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:52.287747 2579 patch_prober.go:28] interesting pod/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 30 15:59:52.288232 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:52.287830 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" podUID="1da12581-0837-4ada-ab71-9dfb216cb4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 30 15:59:53.287234 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:53.287198 2579 patch_prober.go:28] interesting pod/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 30 15:59:53.287425 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:53.287265 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" podUID="1da12581-0837-4ada-ab71-9dfb216cb4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 30 15:59:54.290858 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:54.290826 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:54.291333 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:54.291090 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 15:59:54.291710 ip-10-0-142-244 kubenswrapper[2579]: I0730 15:59:54.291695 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 16:00:16.056267 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.056234 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn"] Jul 30 16:00:16.056778 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.056515 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn" podUID="1d17487c-7cae-4ab3-b8d8-733d0bab643e" containerName="payload-processing" containerID="cri-o://52e339017b9f93fd06dc70956da53b68fa59651d7b16234d1d3083f69788d77a" gracePeriod=30 Jul 30 16:00:16.063748 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.063719 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq"] Jul 30 16:00:16.064524 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.064491 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" podUID="96010de1-983a-4e58-b517-6ee77b6fc760" containerName="payload-pre-processing" containerID="cri-o://352281391acbfb5e2e82352a69844a9e845ab57530adf7dc6d32e70f2e486e9f" gracePeriod=30 Jul 30 16:00:16.318814 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.318746 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn" Jul 30 16:00:16.351217 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.351191 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" Jul 30 16:00:16.412196 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.412170 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76p5x\" (UniqueName: \"kubernetes.io/projected/96010de1-983a-4e58-b517-6ee77b6fc760-kube-api-access-76p5x\") pod \"96010de1-983a-4e58-b517-6ee77b6fc760\" (UID: \"96010de1-983a-4e58-b517-6ee77b6fc760\") " Jul 30 16:00:16.412309 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.412223 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96010de1-983a-4e58-b517-6ee77b6fc760-plugins-config-volume\") pod \"96010de1-983a-4e58-b517-6ee77b6fc760\" (UID: \"96010de1-983a-4e58-b517-6ee77b6fc760\") " Jul 30 16:00:16.412309 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.412261 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbvbh\" (UniqueName: \"kubernetes.io/projected/1d17487c-7cae-4ab3-b8d8-733d0bab643e-kube-api-access-hbvbh\") pod \"1d17487c-7cae-4ab3-b8d8-733d0bab643e\" (UID: \"1d17487c-7cae-4ab3-b8d8-733d0bab643e\") " Jul 30 16:00:16.412309 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.412279 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d17487c-7cae-4ab3-b8d8-733d0bab643e-plugins-config-volume\") pod \"1d17487c-7cae-4ab3-b8d8-733d0bab643e\" (UID: \"1d17487c-7cae-4ab3-b8d8-733d0bab643e\") " Jul 30 16:00:16.412659 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.412632 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96010de1-983a-4e58-b517-6ee77b6fc760-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96010de1-983a-4e58-b517-6ee77b6fc760" (UID: "96010de1-983a-4e58-b517-6ee77b6fc760"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 16:00:16.412750 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.412674 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d17487c-7cae-4ab3-b8d8-733d0bab643e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d17487c-7cae-4ab3-b8d8-733d0bab643e" (UID: "1d17487c-7cae-4ab3-b8d8-733d0bab643e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 16:00:16.414172 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.414152 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96010de1-983a-4e58-b517-6ee77b6fc760-kube-api-access-76p5x" (OuterVolumeSpecName: "kube-api-access-76p5x") pod "96010de1-983a-4e58-b517-6ee77b6fc760" (UID: "96010de1-983a-4e58-b517-6ee77b6fc760"). InnerVolumeSpecName "kube-api-access-76p5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:00:16.414257 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.414213 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d17487c-7cae-4ab3-b8d8-733d0bab643e-kube-api-access-hbvbh" (OuterVolumeSpecName: "kube-api-access-hbvbh") pod "1d17487c-7cae-4ab3-b8d8-733d0bab643e" (UID: "1d17487c-7cae-4ab3-b8d8-733d0bab643e"). InnerVolumeSpecName "kube-api-access-hbvbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:00:16.513212 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.513186 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbvbh\" (UniqueName: \"kubernetes.io/projected/1d17487c-7cae-4ab3-b8d8-733d0bab643e-kube-api-access-hbvbh\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:16.513212 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.513211 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d17487c-7cae-4ab3-b8d8-733d0bab643e-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:16.513373 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.513224 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76p5x\" (UniqueName: \"kubernetes.io/projected/96010de1-983a-4e58-b517-6ee77b6fc760-kube-api-access-76p5x\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:16.513373 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.513237 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96010de1-983a-4e58-b517-6ee77b6fc760-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:16.550253 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.550227 2579 generic.go:356] "Generic (PLEG): container finished" podID="96010de1-983a-4e58-b517-6ee77b6fc760" containerID="352281391acbfb5e2e82352a69844a9e845ab57530adf7dc6d32e70f2e486e9f" exitCode=0 Jul 30 16:00:16.550364 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.550274 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" Jul 30 16:00:16.550364 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.550308 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" event={"ID":"96010de1-983a-4e58-b517-6ee77b6fc760","Type":"ContainerDied","Data":"352281391acbfb5e2e82352a69844a9e845ab57530adf7dc6d32e70f2e486e9f"} Jul 30 16:00:16.550364 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.550359 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq" event={"ID":"96010de1-983a-4e58-b517-6ee77b6fc760","Type":"ContainerDied","Data":"7aabdd83c4a62c6a0cd2c80fba2a5e8a4ac65697d06efea5ba03fcdc9c77e77a"} Jul 30 16:00:16.550515 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.550389 2579 scope.go:117] "RemoveContainer" containerID="352281391acbfb5e2e82352a69844a9e845ab57530adf7dc6d32e70f2e486e9f" Jul 30 16:00:16.551770 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.551746 2579 generic.go:356] "Generic (PLEG): container finished" podID="1d17487c-7cae-4ab3-b8d8-733d0bab643e" containerID="52e339017b9f93fd06dc70956da53b68fa59651d7b16234d1d3083f69788d77a" exitCode=0 Jul 30 16:00:16.551905 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.551788 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn" event={"ID":"1d17487c-7cae-4ab3-b8d8-733d0bab643e","Type":"ContainerDied","Data":"52e339017b9f93fd06dc70956da53b68fa59651d7b16234d1d3083f69788d77a"} Jul 30 16:00:16.551905 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.551817 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn" Jul 30 16:00:16.551905 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.551853 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn" event={"ID":"1d17487c-7cae-4ab3-b8d8-733d0bab643e","Type":"ContainerDied","Data":"28697911c1f388b15aae72c61d6d8e1e8217a5407460a07f1ef3e73412c088b6"} Jul 30 16:00:16.560261 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.560244 2579 scope.go:117] "RemoveContainer" containerID="352281391acbfb5e2e82352a69844a9e845ab57530adf7dc6d32e70f2e486e9f" Jul 30 16:00:16.560544 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:00:16.560525 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352281391acbfb5e2e82352a69844a9e845ab57530adf7dc6d32e70f2e486e9f\": container with ID starting with 352281391acbfb5e2e82352a69844a9e845ab57530adf7dc6d32e70f2e486e9f not found: ID does not exist" containerID="352281391acbfb5e2e82352a69844a9e845ab57530adf7dc6d32e70f2e486e9f" Jul 30 16:00:16.560605 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.560552 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352281391acbfb5e2e82352a69844a9e845ab57530adf7dc6d32e70f2e486e9f"} err="failed to get container status \"352281391acbfb5e2e82352a69844a9e845ab57530adf7dc6d32e70f2e486e9f\": rpc error: code = NotFound desc = could not find container \"352281391acbfb5e2e82352a69844a9e845ab57530adf7dc6d32e70f2e486e9f\": container with ID starting with 352281391acbfb5e2e82352a69844a9e845ab57530adf7dc6d32e70f2e486e9f not found: ID does not exist" Jul 30 16:00:16.560605 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.560569 2579 scope.go:117] "RemoveContainer" containerID="52e339017b9f93fd06dc70956da53b68fa59651d7b16234d1d3083f69788d77a" Jul 30 16:00:16.568404 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.568386 2579 scope.go:117] "RemoveContainer" containerID="52e339017b9f93fd06dc70956da53b68fa59651d7b16234d1d3083f69788d77a" Jul 30 16:00:16.568617 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:00:16.568598 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52e339017b9f93fd06dc70956da53b68fa59651d7b16234d1d3083f69788d77a\": container with ID starting with 52e339017b9f93fd06dc70956da53b68fa59651d7b16234d1d3083f69788d77a not found: ID does not exist" containerID="52e339017b9f93fd06dc70956da53b68fa59651d7b16234d1d3083f69788d77a" Jul 30 16:00:16.568667 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.568623 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52e339017b9f93fd06dc70956da53b68fa59651d7b16234d1d3083f69788d77a"} err="failed to get container status \"52e339017b9f93fd06dc70956da53b68fa59651d7b16234d1d3083f69788d77a\": rpc error: code = NotFound desc = could not find container \"52e339017b9f93fd06dc70956da53b68fa59651d7b16234d1d3083f69788d77a\": container with ID starting with 52e339017b9f93fd06dc70956da53b68fa59651d7b16234d1d3083f69788d77a not found: ID does not exist" Jul 30 16:00:16.575054 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.575003 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq"] Jul 30 16:00:16.582043 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.582012 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-51545f-5c5b97cd4-jwkrq"] Jul 30 16:00:16.593479 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.593459 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn"] Jul 30 16:00:16.597705 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:16.597687 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-51545f-85f79b56b8-wzcrn"] Jul 30 16:00:17.198302 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:17.198267 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d17487c-7cae-4ab3-b8d8-733d0bab643e" path="/var/lib/kubelet/pods/1d17487c-7cae-4ab3-b8d8-733d0bab643e/volumes" Jul 30 16:00:17.198705 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:17.198659 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96010de1-983a-4e58-b517-6ee77b6fc760" path="/var/lib/kubelet/pods/96010de1-983a-4e58-b517-6ee77b6fc760/volumes" Jul 30 16:00:18.529733 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:18.529703 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:00:22.435024 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:22.434992 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr"] Jul 30 16:00:22.435605 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:22.435215 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" podUID="f89b1c19-f279-4d41-bbc8-df322e5c1986" containerName="istio-proxy" containerID="cri-o://276b66cf5d15c04bfe271827a88ea724a0f59fc614e2b86b35dc547e28191a36" gracePeriod=30 Jul 30 16:00:27.610535 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.610509 2579 generic.go:356] "Generic (PLEG): container finished" podID="f89b1c19-f279-4d41-bbc8-df322e5c1986" containerID="276b66cf5d15c04bfe271827a88ea724a0f59fc614e2b86b35dc547e28191a36" exitCode=0 Jul 30 16:00:27.610880 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.610582 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" event={"ID":"f89b1c19-f279-4d41-bbc8-df322e5c1986","Type":"ContainerDied","Data":"276b66cf5d15c04bfe271827a88ea724a0f59fc614e2b86b35dc547e28191a36"} Jul 30 16:00:27.610880 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.610651 2579 scope.go:117] "RemoveContainer" containerID="45ebcb0227fd0b03979c67eef166948e441f3fbea68468e5604c927fa050bb66" Jul 30 16:00:27.689666 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.689644 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 16:00:27.818660 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.818550 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-credential-socket\") pod \"f89b1c19-f279-4d41-bbc8-df322e5c1986\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " Jul 30 16:00:27.818660 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.818593 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlvfc\" (UniqueName: \"kubernetes.io/projected/f89b1c19-f279-4d41-bbc8-df322e5c1986-kube-api-access-vlvfc\") pod \"f89b1c19-f279-4d41-bbc8-df322e5c1986\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " Jul 30 16:00:27.818660 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.818622 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89b1c19-f279-4d41-bbc8-df322e5c1986-istiod-ca-cert\") pod \"f89b1c19-f279-4d41-bbc8-df322e5c1986\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " Jul 30 16:00:27.818973 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.818781 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-envoy\") pod \"f89b1c19-f279-4d41-bbc8-df322e5c1986\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " Jul 30 16:00:27.818973 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.818858 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-workload-certs\") pod \"f89b1c19-f279-4d41-bbc8-df322e5c1986\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " Jul 30 16:00:27.818973 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.818873 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f89b1c19-f279-4d41-bbc8-df322e5c1986" (UID: "f89b1c19-f279-4d41-bbc8-df322e5c1986"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:00:27.818973 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.818922 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-workload-socket\") pod \"f89b1c19-f279-4d41-bbc8-df322e5c1986\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " Jul 30 16:00:27.818973 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.818964 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-token\") pod \"f89b1c19-f279-4d41-bbc8-df322e5c1986\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " Jul 30 16:00:27.819228 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.819006 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89b1c19-f279-4d41-bbc8-df322e5c1986-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f89b1c19-f279-4d41-bbc8-df322e5c1986" (UID: "f89b1c19-f279-4d41-bbc8-df322e5c1986"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 16:00:27.819228 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.819039 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-podinfo\") pod \"f89b1c19-f279-4d41-bbc8-df322e5c1986\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " Jul 30 16:00:27.819228 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.819077 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-data\") pod \"f89b1c19-f279-4d41-bbc8-df322e5c1986\" (UID: \"f89b1c19-f279-4d41-bbc8-df322e5c1986\") " Jul 30 16:00:27.819228 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.819126 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f89b1c19-f279-4d41-bbc8-df322e5c1986" (UID: "f89b1c19-f279-4d41-bbc8-df322e5c1986"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:00:27.819228 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.819168 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f89b1c19-f279-4d41-bbc8-df322e5c1986" (UID: "f89b1c19-f279-4d41-bbc8-df322e5c1986"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:00:27.819491 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.819471 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89b1c19-f279-4d41-bbc8-df322e5c1986-istiod-ca-cert\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:27.819554 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.819494 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-workload-certs\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:27.819554 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.819502 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-workload-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:27.819554 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.819514 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-credential-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:27.820079 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.820052 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-data" (OuterVolumeSpecName: "istio-data") pod "f89b1c19-f279-4d41-bbc8-df322e5c1986" (UID: "f89b1c19-f279-4d41-bbc8-df322e5c1986"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:00:27.821046 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.821018 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f89b1c19-f279-4d41-bbc8-df322e5c1986" (UID: "f89b1c19-f279-4d41-bbc8-df322e5c1986"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:00:27.821206 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.821182 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f89b1c19-f279-4d41-bbc8-df322e5c1986" (UID: "f89b1c19-f279-4d41-bbc8-df322e5c1986"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 16:00:27.821308 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.821278 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-token" (OuterVolumeSpecName: "istio-token") pod "f89b1c19-f279-4d41-bbc8-df322e5c1986" (UID: "f89b1c19-f279-4d41-bbc8-df322e5c1986"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:00:27.821364 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.821298 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89b1c19-f279-4d41-bbc8-df322e5c1986-kube-api-access-vlvfc" (OuterVolumeSpecName: "kube-api-access-vlvfc") pod "f89b1c19-f279-4d41-bbc8-df322e5c1986" (UID: "f89b1c19-f279-4d41-bbc8-df322e5c1986"). InnerVolumeSpecName "kube-api-access-vlvfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:00:27.920851 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.920821 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-envoy\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:27.920851 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.920845 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-token\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:27.920851 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.920854 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-podinfo\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:27.921053 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.920863 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89b1c19-f279-4d41-bbc8-df322e5c1986-istio-data\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:27.921053 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:27.920871 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlvfc\" (UniqueName: \"kubernetes.io/projected/f89b1c19-f279-4d41-bbc8-df322e5c1986-kube-api-access-vlvfc\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:28.622524 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:28.622492 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" Jul 30 16:00:28.622946 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:28.622522 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr" event={"ID":"f89b1c19-f279-4d41-bbc8-df322e5c1986","Type":"ContainerDied","Data":"75a4c097699f5179cb492b0e99e56838a58c9bb0cf57c926ac6dcbfe306fa6cf"} Jul 30 16:00:28.622946 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:28.622606 2579 scope.go:117] "RemoveContainer" containerID="276b66cf5d15c04bfe271827a88ea724a0f59fc614e2b86b35dc547e28191a36" Jul 30 16:00:28.645253 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:28.645229 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr"] Jul 30 16:00:28.648491 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:28.648470 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-51545f-openshift-default-5b99c567c6-828wr"] Jul 30 16:00:29.196911 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:29.196880 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f89b1c19-f279-4d41-bbc8-df322e5c1986" path="/var/lib/kubelet/pods/f89b1c19-f279-4d41-bbc8-df322e5c1986/volumes" Jul 30 16:00:38.253192 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.253156 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg"] Jul 30 16:00:38.253673 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.253439 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg" podUID="d88f484e-a906-4390-aa5c-7864115a2d9a" containerName="payload-processing" containerID="cri-o://5b873bff2fc8cb9e976da979ff993ed0fae84e81a878d1ac67796def0e2ae66c" gracePeriod=30 Jul 30 16:00:38.261225 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.261196 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt"] Jul 30 16:00:38.261448 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.261418 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt" podUID="d98e5399-d19b-4b38-8cb0-0921222abced" containerName="payload-pre-processing" containerID="cri-o://46fd10c03228691a14c20844aeaf5ad2ca2ac94a0a686e5ca89daad9e9b8e0e2" gracePeriod=30 Jul 30 16:00:38.512048 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.511993 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg" Jul 30 16:00:38.548934 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.548905 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt" Jul 30 16:00:38.616858 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.616827 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfw64\" (UniqueName: \"kubernetes.io/projected/d88f484e-a906-4390-aa5c-7864115a2d9a-kube-api-access-tfw64\") pod \"d88f484e-a906-4390-aa5c-7864115a2d9a\" (UID: \"d88f484e-a906-4390-aa5c-7864115a2d9a\") " Jul 30 16:00:38.616996 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.616934 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88f484e-a906-4390-aa5c-7864115a2d9a-plugins-config-volume\") pod \"d88f484e-a906-4390-aa5c-7864115a2d9a\" (UID: \"d88f484e-a906-4390-aa5c-7864115a2d9a\") " Jul 30 16:00:38.617293 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.617269 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d88f484e-a906-4390-aa5c-7864115a2d9a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d88f484e-a906-4390-aa5c-7864115a2d9a" (UID: "d88f484e-a906-4390-aa5c-7864115a2d9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 16:00:38.618784 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.618763 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88f484e-a906-4390-aa5c-7864115a2d9a-kube-api-access-tfw64" (OuterVolumeSpecName: "kube-api-access-tfw64") pod "d88f484e-a906-4390-aa5c-7864115a2d9a" (UID: "d88f484e-a906-4390-aa5c-7864115a2d9a"). InnerVolumeSpecName "kube-api-access-tfw64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:00:38.671742 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.671706 2579 generic.go:356] "Generic (PLEG): container finished" podID="d88f484e-a906-4390-aa5c-7864115a2d9a" containerID="5b873bff2fc8cb9e976da979ff993ed0fae84e81a878d1ac67796def0e2ae66c" exitCode=0 Jul 30 16:00:38.671921 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.671751 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg" Jul 30 16:00:38.671921 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.671790 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg" event={"ID":"d88f484e-a906-4390-aa5c-7864115a2d9a","Type":"ContainerDied","Data":"5b873bff2fc8cb9e976da979ff993ed0fae84e81a878d1ac67796def0e2ae66c"} Jul 30 16:00:38.671921 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.671870 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg" event={"ID":"d88f484e-a906-4390-aa5c-7864115a2d9a","Type":"ContainerDied","Data":"26491c06a5939245e2a42ec7c9f2340a49429dd15fb83bfb521f427dcafe42f2"} Jul 30 16:00:38.671921 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.671894 2579 scope.go:117] "RemoveContainer" containerID="5b873bff2fc8cb9e976da979ff993ed0fae84e81a878d1ac67796def0e2ae66c" Jul 30 16:00:38.672998 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.672979 2579 generic.go:356] "Generic (PLEG): container finished" podID="d98e5399-d19b-4b38-8cb0-0921222abced" containerID="46fd10c03228691a14c20844aeaf5ad2ca2ac94a0a686e5ca89daad9e9b8e0e2" exitCode=0 Jul 30 16:00:38.673087 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.673005 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt" event={"ID":"d98e5399-d19b-4b38-8cb0-0921222abced","Type":"ContainerDied","Data":"46fd10c03228691a14c20844aeaf5ad2ca2ac94a0a686e5ca89daad9e9b8e0e2"} Jul 30 16:00:38.673087 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.673036 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt" Jul 30 16:00:38.673087 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.673050 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt" event={"ID":"d98e5399-d19b-4b38-8cb0-0921222abced","Type":"ContainerDied","Data":"d8a54066739ff7201903a2d37f4bbd2f4189f31b38e7a1bb7f2b17b1ddeee50b"} Jul 30 16:00:38.681237 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.681220 2579 scope.go:117] "RemoveContainer" containerID="5b873bff2fc8cb9e976da979ff993ed0fae84e81a878d1ac67796def0e2ae66c" Jul 30 16:00:38.681492 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:00:38.681470 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b873bff2fc8cb9e976da979ff993ed0fae84e81a878d1ac67796def0e2ae66c\": container with ID starting with 5b873bff2fc8cb9e976da979ff993ed0fae84e81a878d1ac67796def0e2ae66c not found: ID does not exist" containerID="5b873bff2fc8cb9e976da979ff993ed0fae84e81a878d1ac67796def0e2ae66c" Jul 30 16:00:38.681576 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.681497 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b873bff2fc8cb9e976da979ff993ed0fae84e81a878d1ac67796def0e2ae66c"} err="failed to get container status \"5b873bff2fc8cb9e976da979ff993ed0fae84e81a878d1ac67796def0e2ae66c\": rpc error: code = NotFound desc = could not find container \"5b873bff2fc8cb9e976da979ff993ed0fae84e81a878d1ac67796def0e2ae66c\": container with ID starting with 5b873bff2fc8cb9e976da979ff993ed0fae84e81a878d1ac67796def0e2ae66c not found: ID does not exist" Jul 30 16:00:38.681576 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.681512 2579 scope.go:117] "RemoveContainer" containerID="46fd10c03228691a14c20844aeaf5ad2ca2ac94a0a686e5ca89daad9e9b8e0e2" Jul 30 16:00:38.689063 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.689045 2579 scope.go:117] "RemoveContainer" containerID="46fd10c03228691a14c20844aeaf5ad2ca2ac94a0a686e5ca89daad9e9b8e0e2" Jul 30 16:00:38.689314 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:00:38.689291 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46fd10c03228691a14c20844aeaf5ad2ca2ac94a0a686e5ca89daad9e9b8e0e2\": container with ID starting with 46fd10c03228691a14c20844aeaf5ad2ca2ac94a0a686e5ca89daad9e9b8e0e2 not found: ID does not exist" containerID="46fd10c03228691a14c20844aeaf5ad2ca2ac94a0a686e5ca89daad9e9b8e0e2" Jul 30 16:00:38.689410 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.689319 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46fd10c03228691a14c20844aeaf5ad2ca2ac94a0a686e5ca89daad9e9b8e0e2"} err="failed to get container status \"46fd10c03228691a14c20844aeaf5ad2ca2ac94a0a686e5ca89daad9e9b8e0e2\": rpc error: code = NotFound desc = could not find container \"46fd10c03228691a14c20844aeaf5ad2ca2ac94a0a686e5ca89daad9e9b8e0e2\": container with ID starting with 46fd10c03228691a14c20844aeaf5ad2ca2ac94a0a686e5ca89daad9e9b8e0e2 not found: ID does not exist" Jul 30 16:00:38.693951 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.693931 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg"] Jul 30 16:00:38.699491 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.699472 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-985263-9945dfdbd-h2bwg"] Jul 30 16:00:38.718062 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.718043 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n28j\" (UniqueName: \"kubernetes.io/projected/d98e5399-d19b-4b38-8cb0-0921222abced-kube-api-access-8n28j\") pod \"d98e5399-d19b-4b38-8cb0-0921222abced\" (UID: \"d98e5399-d19b-4b38-8cb0-0921222abced\") " Jul 30 16:00:38.718143 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.718078 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98e5399-d19b-4b38-8cb0-0921222abced-plugins-config-volume\") pod \"d98e5399-d19b-4b38-8cb0-0921222abced\" (UID: \"d98e5399-d19b-4b38-8cb0-0921222abced\") " Jul 30 16:00:38.718287 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.718275 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88f484e-a906-4390-aa5c-7864115a2d9a-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:38.718353 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.718287 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfw64\" (UniqueName: \"kubernetes.io/projected/d88f484e-a906-4390-aa5c-7864115a2d9a-kube-api-access-tfw64\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:38.718476 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.718456 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98e5399-d19b-4b38-8cb0-0921222abced-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d98e5399-d19b-4b38-8cb0-0921222abced" (UID: "d98e5399-d19b-4b38-8cb0-0921222abced"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 16:00:38.719863 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.719841 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98e5399-d19b-4b38-8cb0-0921222abced-kube-api-access-8n28j" (OuterVolumeSpecName: "kube-api-access-8n28j") pod "d98e5399-d19b-4b38-8cb0-0921222abced" (UID: "d98e5399-d19b-4b38-8cb0-0921222abced"). InnerVolumeSpecName "kube-api-access-8n28j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:00:38.819033 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.818949 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98e5399-d19b-4b38-8cb0-0921222abced-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:38.819033 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.818984 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n28j\" (UniqueName: \"kubernetes.io/projected/d98e5399-d19b-4b38-8cb0-0921222abced-kube-api-access-8n28j\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:38.998982 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:38.998949 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt"] Jul 30 16:00:39.002957 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:39.002934 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-985263-558747c468-sf8pt"] Jul 30 16:00:39.198123 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:39.198086 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d88f484e-a906-4390-aa5c-7864115a2d9a" path="/var/lib/kubelet/pods/d88f484e-a906-4390-aa5c-7864115a2d9a/volumes" Jul 30 16:00:39.198468 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:39.198455 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d98e5399-d19b-4b38-8cb0-0921222abced" path="/var/lib/kubelet/pods/d98e5399-d19b-4b38-8cb0-0921222abced/volumes" Jul 30 16:00:43.337679 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:43.337641 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:00:44.812013 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:44.811965 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9"] Jul 30 16:00:44.812473 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:44.812223 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" podUID="1da12581-0837-4ada-ab71-9dfb216cb4fc" containerName="istio-proxy" containerID="cri-o://d110946e2c8d6d441691b9eab84aa933b7a5a874505424e16eccd90afba29880" gracePeriod=30 Jul 30 16:00:45.734555 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.734520 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs"] Jul 30 16:00:45.735381 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.735360 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f89b1c19-f279-4d41-bbc8-df322e5c1986" containerName="istio-proxy" Jul 30 16:00:45.735517 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.735506 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f89b1c19-f279-4d41-bbc8-df322e5c1986" containerName="istio-proxy" Jul 30 16:00:45.735608 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.735598 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d88f484e-a906-4390-aa5c-7864115a2d9a" containerName="payload-processing" Jul 30 16:00:45.735684 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.735675 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d88f484e-a906-4390-aa5c-7864115a2d9a" containerName="payload-processing" Jul 30 16:00:45.735757 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.735747 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96010de1-983a-4e58-b517-6ee77b6fc760" containerName="payload-pre-processing" Jul 30 16:00:45.735844 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.735834 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="96010de1-983a-4e58-b517-6ee77b6fc760" containerName="payload-pre-processing" Jul 30 16:00:45.735925 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.735916 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d98e5399-d19b-4b38-8cb0-0921222abced" containerName="payload-pre-processing" Jul 30 16:00:45.735988 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.735980 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d98e5399-d19b-4b38-8cb0-0921222abced" containerName="payload-pre-processing" Jul 30 16:00:45.736066 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.736057 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f89b1c19-f279-4d41-bbc8-df322e5c1986" containerName="istio-proxy" Jul 30 16:00:45.736137 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.736130 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f89b1c19-f279-4d41-bbc8-df322e5c1986" containerName="istio-proxy" Jul 30 16:00:45.736227 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.736218 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d17487c-7cae-4ab3-b8d8-733d0bab643e" containerName="payload-processing" Jul 30 16:00:45.736293 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.736285 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d17487c-7cae-4ab3-b8d8-733d0bab643e" containerName="payload-processing" Jul 30 16:00:45.736453 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.736443 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d17487c-7cae-4ab3-b8d8-733d0bab643e" containerName="payload-processing" Jul 30 16:00:45.736529 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.736518 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d98e5399-d19b-4b38-8cb0-0921222abced" containerName="payload-pre-processing" Jul 30 16:00:45.736600 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.736592 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="96010de1-983a-4e58-b517-6ee77b6fc760" containerName="payload-pre-processing" Jul 30 16:00:45.736667 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.736659 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f89b1c19-f279-4d41-bbc8-df322e5c1986" containerName="istio-proxy" Jul 30 16:00:45.736818 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.736790 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d88f484e-a906-4390-aa5c-7864115a2d9a" containerName="payload-processing" Jul 30 16:00:45.745742 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.745717 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.748410 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.748392 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-f2cf4d-openshift-default-dockercfg-wrtn2\"" Jul 30 16:00:45.748701 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.748674 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs"] Jul 30 16:00:45.777241 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.777206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-workload-certs\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.777241 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.777242 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-envoy\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.777445 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.777268 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmlcz\" (UniqueName: \"kubernetes.io/projected/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-kube-api-access-pmlcz\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.777445 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.777342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-podinfo\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.777445 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.777378 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-credential-socket\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.777445 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.777427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-data\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.777576 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.777453 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-token\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.777576 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.777489 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istiod-ca-cert\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.777576 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.777556 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-workload-socket\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.878599 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.878564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-workload-socket\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.879067 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.878607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-workload-certs\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.879067 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.878627 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-envoy\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.879067 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.878662 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmlcz\" (UniqueName: \"kubernetes.io/projected/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-kube-api-access-pmlcz\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.879067 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.878716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-podinfo\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.879067 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.878752 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-credential-socket\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.879067 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.878791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-data\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.879067 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.878847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-token\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.879067 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.878883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istiod-ca-cert\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.879465 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.879161 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-workload-certs\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.879465 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.879201 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-data\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.879465 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.879253 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-credential-socket\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.879465 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.879367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-workload-socket\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.879600 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.879529 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istiod-ca-cert\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.880965 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.880938 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-envoy\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.881116 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.881096 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-podinfo\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.888123 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.888095 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-token\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:45.888238 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:45.888221 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmlcz\" (UniqueName: \"kubernetes.io/projected/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-kube-api-access-pmlcz\") pod \"e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:46.057826 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:46.057718 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:46.198973 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:46.198945 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs"] Jul 30 16:00:46.200223 ip-10-0-142-244 kubenswrapper[2579]: W0730 16:00:46.200198 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb02030_4a7c_4a7f_aa1f_a5308fe128c4.slice/crio-2892c93dd07ee8c69374fde9e13ad490dce6f3f34ba4c85dd654669c1755ac27 WatchSource:0}: Error finding container 2892c93dd07ee8c69374fde9e13ad490dce6f3f34ba4c85dd654669c1755ac27: Status 404 returned error can't find the container with id 2892c93dd07ee8c69374fde9e13ad490dce6f3f34ba4c85dd654669c1755ac27 Jul 30 16:00:46.202294 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:46.202263 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 16:00:46.202363 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:46.202324 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 16:00:46.202363 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:46.202353 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 16:00:46.715066 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:46.715028 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" event={"ID":"afb02030-4a7c-4a7f-aa1f-a5308fe128c4","Type":"ContainerStarted","Data":"1211e8cf0590ed48f5664d96b6e16282b6b4ec924f1abf9a451a27f6ab317bdb"} Jul 30 16:00:46.715250 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:46.715090 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" event={"ID":"afb02030-4a7c-4a7f-aa1f-a5308fe128c4","Type":"ContainerStarted","Data":"2892c93dd07ee8c69374fde9e13ad490dce6f3f34ba4c85dd654669c1755ac27"} Jul 30 16:00:46.735638 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:46.735589 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" podStartSLOduration=1.735574699 podStartE2EDuration="1.735574699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 16:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 16:00:46.733314717 +0000 UTC m=+3073.968445562" watchObservedRunningTime="2026-07-30 16:00:46.735574699 +0000 UTC m=+3073.970705544" Jul 30 16:00:47.058913 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:47.058834 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:47.063662 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:47.063639 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:47.720915 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:47.720891 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:47.721754 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:47.721736 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:00:50.069050 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.069023 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 16:00:50.120723 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.120697 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-podinfo\") pod \"1da12581-0837-4ada-ab71-9dfb216cb4fc\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " Jul 30 16:00:50.120912 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.120755 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-envoy\") pod \"1da12581-0837-4ada-ab71-9dfb216cb4fc\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " Jul 30 16:00:50.120912 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.120818 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-data\") pod \"1da12581-0837-4ada-ab71-9dfb216cb4fc\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " Jul 30 16:00:50.120912 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.120870 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-token\") pod \"1da12581-0837-4ada-ab71-9dfb216cb4fc\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " Jul 30 16:00:50.120912 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.120910 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2b6f\" (UniqueName: \"kubernetes.io/projected/1da12581-0837-4ada-ab71-9dfb216cb4fc-kube-api-access-q2b6f\") pod \"1da12581-0837-4ada-ab71-9dfb216cb4fc\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " Jul 30 16:00:50.121152 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.120953 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-credential-socket\") pod \"1da12581-0837-4ada-ab71-9dfb216cb4fc\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " Jul 30 16:00:50.121152 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.120985 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-workload-certs\") pod \"1da12581-0837-4ada-ab71-9dfb216cb4fc\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " Jul 30 16:00:50.121152 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.121020 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1da12581-0837-4ada-ab71-9dfb216cb4fc-istiod-ca-cert\") pod \"1da12581-0837-4ada-ab71-9dfb216cb4fc\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " Jul 30 16:00:50.121152 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.121077 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-workload-socket\") pod \"1da12581-0837-4ada-ab71-9dfb216cb4fc\" (UID: \"1da12581-0837-4ada-ab71-9dfb216cb4fc\") " Jul 30 16:00:50.121367 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.121225 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1da12581-0837-4ada-ab71-9dfb216cb4fc" (UID: "1da12581-0837-4ada-ab71-9dfb216cb4fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:00:50.121367 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.121254 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1da12581-0837-4ada-ab71-9dfb216cb4fc" (UID: "1da12581-0837-4ada-ab71-9dfb216cb4fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:00:50.121500 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.121480 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-credential-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:50.121566 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.121475 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1da12581-0837-4ada-ab71-9dfb216cb4fc" (UID: "1da12581-0837-4ada-ab71-9dfb216cb4fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:00:50.121566 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.121506 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-workload-certs\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:50.121662 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.121588 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da12581-0837-4ada-ab71-9dfb216cb4fc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1da12581-0837-4ada-ab71-9dfb216cb4fc" (UID: "1da12581-0837-4ada-ab71-9dfb216cb4fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 16:00:50.122072 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.122046 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-data" (OuterVolumeSpecName: "istio-data") pod "1da12581-0837-4ada-ab71-9dfb216cb4fc" (UID: "1da12581-0837-4ada-ab71-9dfb216cb4fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:00:50.123008 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.122980 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1da12581-0837-4ada-ab71-9dfb216cb4fc" (UID: "1da12581-0837-4ada-ab71-9dfb216cb4fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 16:00:50.123122 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.123101 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da12581-0837-4ada-ab71-9dfb216cb4fc-kube-api-access-q2b6f" (OuterVolumeSpecName: "kube-api-access-q2b6f") pod "1da12581-0837-4ada-ab71-9dfb216cb4fc" (UID: "1da12581-0837-4ada-ab71-9dfb216cb4fc"). InnerVolumeSpecName "kube-api-access-q2b6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:00:50.123122 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.123107 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1da12581-0837-4ada-ab71-9dfb216cb4fc" (UID: "1da12581-0837-4ada-ab71-9dfb216cb4fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:00:50.123274 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.123259 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-token" (OuterVolumeSpecName: "istio-token") pod "1da12581-0837-4ada-ab71-9dfb216cb4fc" (UID: "1da12581-0837-4ada-ab71-9dfb216cb4fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:00:50.222296 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.222273 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1da12581-0837-4ada-ab71-9dfb216cb4fc-istiod-ca-cert\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:50.222296 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.222298 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-workload-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:50.222435 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.222311 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-podinfo\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:50.222435 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.222320 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-envoy\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:50.222435 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.222330 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-data\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:50.222435 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.222337 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1da12581-0837-4ada-ab71-9dfb216cb4fc-istio-token\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:50.222435 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.222345 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2b6f\" (UniqueName: \"kubernetes.io/projected/1da12581-0837-4ada-ab71-9dfb216cb4fc-kube-api-access-q2b6f\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:00:50.393411 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.393322 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm"] Jul 30 16:00:50.393711 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.393698 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da12581-0837-4ada-ab71-9dfb216cb4fc" containerName="istio-proxy" Jul 30 16:00:50.393755 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.393713 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da12581-0837-4ada-ab71-9dfb216cb4fc" containerName="istio-proxy" Jul 30 16:00:50.393810 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.393785 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1da12581-0837-4ada-ab71-9dfb216cb4fc" containerName="istio-proxy" Jul 30 16:00:50.393810 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.393807 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f89b1c19-f279-4d41-bbc8-df322e5c1986" containerName="istio-proxy" Jul 30 16:00:50.393881 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.393816 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1da12581-0837-4ada-ab71-9dfb216cb4fc" containerName="istio-proxy" Jul 30 16:00:50.396386 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.396372 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm" Jul 30 16:00:50.399059 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.399028 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-f2cf4d\"" Jul 30 16:00:50.399205 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.399191 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-f2cf4d-dockercfg-dtlbz\"" Jul 30 16:00:50.412982 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.410594 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm"] Jul 30 16:00:50.423943 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.423916 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzblf\" (UniqueName: \"kubernetes.io/projected/735a5439-4f58-4176-ac4f-ec42abf7ba46-kube-api-access-pzblf\") pod \"payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm\" (UID: \"735a5439-4f58-4176-ac4f-ec42abf7ba46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm" Jul 30 16:00:50.424056 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.423955 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735a5439-4f58-4176-ac4f-ec42abf7ba46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm\" (UID: \"735a5439-4f58-4176-ac4f-ec42abf7ba46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm" Jul 30 16:00:50.428571 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.428548 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2"] Jul 30 16:00:50.428935 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.428919 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da12581-0837-4ada-ab71-9dfb216cb4fc" containerName="istio-proxy" Jul 30 16:00:50.428995 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.428937 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da12581-0837-4ada-ab71-9dfb216cb4fc" containerName="istio-proxy" Jul 30 16:00:50.431226 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.431210 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2" Jul 30 16:00:50.442195 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.442168 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2"] Jul 30 16:00:50.525054 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.525024 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03f2e58b-1567-44e4-8894-3d3ffa33304d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2\" (UID: \"03f2e58b-1567-44e4-8894-3d3ffa33304d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2" Jul 30 16:00:50.525218 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.525125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzblf\" (UniqueName: \"kubernetes.io/projected/735a5439-4f58-4176-ac4f-ec42abf7ba46-kube-api-access-pzblf\") pod \"payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm\" (UID: \"735a5439-4f58-4176-ac4f-ec42abf7ba46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm" Jul 30 16:00:50.525218 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.525160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgjqq\" (UniqueName: \"kubernetes.io/projected/03f2e58b-1567-44e4-8894-3d3ffa33304d-kube-api-access-bgjqq\") pod \"payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2\" (UID: \"03f2e58b-1567-44e4-8894-3d3ffa33304d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2" Jul 30 16:00:50.525218 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.525180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735a5439-4f58-4176-ac4f-ec42abf7ba46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm\" (UID: \"735a5439-4f58-4176-ac4f-ec42abf7ba46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm" Jul 30 16:00:50.525838 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.525821 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735a5439-4f58-4176-ac4f-ec42abf7ba46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm\" (UID: \"735a5439-4f58-4176-ac4f-ec42abf7ba46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm" Jul 30 16:00:50.533835 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.533816 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzblf\" (UniqueName: \"kubernetes.io/projected/735a5439-4f58-4176-ac4f-ec42abf7ba46-kube-api-access-pzblf\") pod \"payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm\" (UID: \"735a5439-4f58-4176-ac4f-ec42abf7ba46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm" Jul 30 16:00:50.626276 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.626251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03f2e58b-1567-44e4-8894-3d3ffa33304d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2\" (UID: \"03f2e58b-1567-44e4-8894-3d3ffa33304d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2" Jul 30 16:00:50.626400 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.626306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgjqq\" (UniqueName: \"kubernetes.io/projected/03f2e58b-1567-44e4-8894-3d3ffa33304d-kube-api-access-bgjqq\") pod \"payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2\" (UID: \"03f2e58b-1567-44e4-8894-3d3ffa33304d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2" Jul 30 16:00:50.626885 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.626868 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03f2e58b-1567-44e4-8894-3d3ffa33304d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2\" (UID: \"03f2e58b-1567-44e4-8894-3d3ffa33304d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2" Jul 30 16:00:50.634225 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.634206 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgjqq\" (UniqueName: \"kubernetes.io/projected/03f2e58b-1567-44e4-8894-3d3ffa33304d-kube-api-access-bgjqq\") pod \"payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2\" (UID: \"03f2e58b-1567-44e4-8894-3d3ffa33304d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2" Jul 30 16:00:50.711493 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.711440 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm" Jul 30 16:00:50.736023 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.735998 2579 generic.go:356] "Generic (PLEG): container finished" podID="1da12581-0837-4ada-ab71-9dfb216cb4fc" containerID="d110946e2c8d6d441691b9eab84aa933b7a5a874505424e16eccd90afba29880" exitCode=0 Jul 30 16:00:50.736122 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.736057 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" Jul 30 16:00:50.736122 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.736077 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" event={"ID":"1da12581-0837-4ada-ab71-9dfb216cb4fc","Type":"ContainerDied","Data":"d110946e2c8d6d441691b9eab84aa933b7a5a874505424e16eccd90afba29880"} Jul 30 16:00:50.736206 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.736128 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9" event={"ID":"1da12581-0837-4ada-ab71-9dfb216cb4fc","Type":"ContainerDied","Data":"833cc89ed1f7a490a4decb00965ddc9f13570af37020c98945c6ecffe093dd19"} Jul 30 16:00:50.736206 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.736150 2579 scope.go:117] "RemoveContainer" containerID="d110946e2c8d6d441691b9eab84aa933b7a5a874505424e16eccd90afba29880" Jul 30 16:00:50.741078 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.741040 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2" Jul 30 16:00:50.745039 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.744995 2579 scope.go:117] "RemoveContainer" containerID="0e51cb5053d5896d474a52502999f4ba1225600c0aac0c865f5da602f72f4b56" Jul 30 16:00:50.757830 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.757784 2579 scope.go:117] "RemoveContainer" containerID="d110946e2c8d6d441691b9eab84aa933b7a5a874505424e16eccd90afba29880" Jul 30 16:00:50.758153 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:00:50.758133 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d110946e2c8d6d441691b9eab84aa933b7a5a874505424e16eccd90afba29880\": container with ID starting with d110946e2c8d6d441691b9eab84aa933b7a5a874505424e16eccd90afba29880 not found: ID does not exist" containerID="d110946e2c8d6d441691b9eab84aa933b7a5a874505424e16eccd90afba29880" Jul 30 16:00:50.758259 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.758166 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d110946e2c8d6d441691b9eab84aa933b7a5a874505424e16eccd90afba29880"} err="failed to get container status \"d110946e2c8d6d441691b9eab84aa933b7a5a874505424e16eccd90afba29880\": rpc error: code = NotFound desc = could not find container \"d110946e2c8d6d441691b9eab84aa933b7a5a874505424e16eccd90afba29880\": container with ID starting with d110946e2c8d6d441691b9eab84aa933b7a5a874505424e16eccd90afba29880 not found: ID does not exist" Jul 30 16:00:50.758259 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.758193 2579 scope.go:117] "RemoveContainer" containerID="0e51cb5053d5896d474a52502999f4ba1225600c0aac0c865f5da602f72f4b56" Jul 30 16:00:50.758490 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:00:50.758470 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e51cb5053d5896d474a52502999f4ba1225600c0aac0c865f5da602f72f4b56\": container with ID starting with 0e51cb5053d5896d474a52502999f4ba1225600c0aac0c865f5da602f72f4b56 not found: ID does not exist" containerID="0e51cb5053d5896d474a52502999f4ba1225600c0aac0c865f5da602f72f4b56" Jul 30 16:00:50.758561 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.758498 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e51cb5053d5896d474a52502999f4ba1225600c0aac0c865f5da602f72f4b56"} err="failed to get container status \"0e51cb5053d5896d474a52502999f4ba1225600c0aac0c865f5da602f72f4b56\": rpc error: code = NotFound desc = could not find container \"0e51cb5053d5896d474a52502999f4ba1225600c0aac0c865f5da602f72f4b56\": container with ID starting with 0e51cb5053d5896d474a52502999f4ba1225600c0aac0c865f5da602f72f4b56 not found: ID does not exist" Jul 30 16:00:50.759381 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.759346 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9"] Jul 30 16:00:50.763874 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.763843 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-985263-openshift-default-6b6ddcd576-955z9"] Jul 30 16:00:50.861501 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.861466 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm"] Jul 30 16:00:50.863281 ip-10-0-142-244 kubenswrapper[2579]: W0730 16:00:50.863251 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735a5439_4f58_4176_ac4f_ec42abf7ba46.slice/crio-36f2345ab5200a302b3e6718b6454669113d362c9a7925e457e27cf920d23c70 WatchSource:0}: Error finding container 36f2345ab5200a302b3e6718b6454669113d362c9a7925e457e27cf920d23c70: Status 404 returned error can't find the container with id 36f2345ab5200a302b3e6718b6454669113d362c9a7925e457e27cf920d23c70 Jul 30 16:00:50.894473 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:50.894446 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2"] Jul 30 16:00:50.902193 ip-10-0-142-244 kubenswrapper[2579]: W0730 16:00:50.902167 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f2e58b_1567_44e4_8894_3d3ffa33304d.slice/crio-591bc80f6feacd86016ce971eeef38d55d7e53cf1e75ce773f922f8546cb9762 WatchSource:0}: Error finding container 591bc80f6feacd86016ce971eeef38d55d7e53cf1e75ce773f922f8546cb9762: Status 404 returned error can't find the container with id 591bc80f6feacd86016ce971eeef38d55d7e53cf1e75ce773f922f8546cb9762 Jul 30 16:00:51.198457 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:51.198423 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1da12581-0837-4ada-ab71-9dfb216cb4fc" path="/var/lib/kubelet/pods/1da12581-0837-4ada-ab71-9dfb216cb4fc/volumes" Jul 30 16:00:51.742968 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:51.742927 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2" event={"ID":"03f2e58b-1567-44e4-8894-3d3ffa33304d","Type":"ContainerStarted","Data":"4783de9b92b21cc3570df21a9f37f7943568afb610cd4b43630d3093ca6f3c11"} Jul 30 16:00:51.743154 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:51.743075 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2" Jul 30 16:00:51.743154 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:51.743119 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2" event={"ID":"03f2e58b-1567-44e4-8894-3d3ffa33304d","Type":"ContainerStarted","Data":"591bc80f6feacd86016ce971eeef38d55d7e53cf1e75ce773f922f8546cb9762"} Jul 30 16:00:51.745190 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:51.745159 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm" event={"ID":"735a5439-4f58-4176-ac4f-ec42abf7ba46","Type":"ContainerStarted","Data":"2296102e577187cbb7f6995192e1fb36e5ac7395ca38e1e97eb8e89deffa1217"} Jul 30 16:00:51.745317 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:51.745201 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm" event={"ID":"735a5439-4f58-4176-ac4f-ec42abf7ba46","Type":"ContainerStarted","Data":"36f2345ab5200a302b3e6718b6454669113d362c9a7925e457e27cf920d23c70"} Jul 30 16:00:51.745317 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:51.745221 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm" Jul 30 16:00:51.761299 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:51.761256 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2" podStartSLOduration=1.761243876 podStartE2EDuration="1.761243876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 16:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 16:00:51.758950581 +0000 UTC m=+3078.994081422" watchObservedRunningTime="2026-07-30 16:00:51.761243876 +0000 UTC m=+3078.996374700" Jul 30 16:00:51.775612 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:51.775573 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm" podStartSLOduration=1.775561611 podStartE2EDuration="1.775561611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 16:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 16:00:51.774400174 +0000 UTC m=+3079.009531020" watchObservedRunningTime="2026-07-30 16:00:51.775561611 +0000 UTC m=+3079.010692456" Jul 30 16:00:54.432872 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:00:54.432838 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:01:02.723156 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:02.723114 2579 patch_prober.go:28] interesting pod/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 30 16:01:02.723573 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:02.723182 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" podUID="afb02030-4a7c-4a7f-aa1f-a5308fe128c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 30 16:01:02.751936 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:02.751905 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm" Jul 30 16:01:02.752076 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:02.751974 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2" Jul 30 16:01:02.810138 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:02.810093 2579 generic.go:356] "Generic (PLEG): container finished" podID="afb02030-4a7c-4a7f-aa1f-a5308fe128c4" containerID="1211e8cf0590ed48f5664d96b6e16282b6b4ec924f1abf9a451a27f6ab317bdb" exitCode=0 Jul 30 16:01:02.810300 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:02.810162 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" event={"ID":"afb02030-4a7c-4a7f-aa1f-a5308fe128c4","Type":"ContainerDied","Data":"1211e8cf0590ed48f5664d96b6e16282b6b4ec924f1abf9a451a27f6ab317bdb"} Jul 30 16:01:02.810625 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:02.810611 2579 scope.go:117] "RemoveContainer" containerID="1211e8cf0590ed48f5664d96b6e16282b6b4ec924f1abf9a451a27f6ab317bdb" Jul 30 16:01:02.811844 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:02.811780 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 16:01:02.811944 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:02.811904 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 16:01:02.811988 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:02.811947 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 16:01:03.817061 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:03.817025 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" event={"ID":"afb02030-4a7c-4a7f-aa1f-a5308fe128c4","Type":"ContainerStarted","Data":"012ff6fa74e1a304f059498f194206143cbc497749e7f7850d4bb28f44ea0130"} Jul 30 16:01:04.058546 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:04.058501 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:01:04.058919 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:04.058889 2579 patch_prober.go:28] interesting pod/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 30 16:01:04.059042 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:04.058955 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" podUID="afb02030-4a7c-4a7f-aa1f-a5308fe128c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 30 16:01:05.059231 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:05.059192 2579 patch_prober.go:28] interesting pod/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 30 16:01:05.059822 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:05.059768 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" podUID="afb02030-4a7c-4a7f-aa1f-a5308fe128c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 30 16:01:06.058732 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:06.058693 2579 patch_prober.go:28] interesting pod/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 30 16:01:06.058961 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:06.058759 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" podUID="afb02030-4a7c-4a7f-aa1f-a5308fe128c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 30 16:01:07.058454 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:07.058410 2579 patch_prober.go:28] interesting pod/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 30 16:01:07.058973 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:07.058522 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" podUID="afb02030-4a7c-4a7f-aa1f-a5308fe128c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 30 16:01:08.062258 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:08.062228 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:01:08.062642 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:08.062483 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:01:08.063180 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:08.063162 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:01:14.460196 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.460162 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td"] Jul 30 16:01:14.463020 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.462999 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.465852 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.465831 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-5468b9-openshift-default-dockercfg-hcvvz\"" Jul 30 16:01:14.474684 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.474659 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td"] Jul 30 16:01:14.562262 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.562218 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-workload-socket\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.562262 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.562269 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-credential-socket\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.562525 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.562294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-token\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.562525 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.562423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-workload-certs\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.562646 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.562525 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brl8d\" (UniqueName: \"kubernetes.io/projected/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-kube-api-access-brl8d\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.562646 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.562561 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-data\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.562646 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.562615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-envoy\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.562646 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.562637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-podinfo\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.562907 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.562665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istiod-ca-cert\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.664100 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.664052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-workload-certs\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.664292 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.664133 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brl8d\" (UniqueName: \"kubernetes.io/projected/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-kube-api-access-brl8d\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.664292 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.664160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-data\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.664292 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.664198 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-envoy\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.664292 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.664230 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-podinfo\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.664292 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.664254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istiod-ca-cert\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.664573 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.664438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-workload-socket\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.664573 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.664484 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-workload-certs\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.664573 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.664519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-credential-socket\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.664573 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.664531 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-data\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.664573 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.664548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-token\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.664870 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.664590 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-workload-socket\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.664870 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.664748 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-credential-socket\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.665070 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.665049 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istiod-ca-cert\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.666530 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.666504 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-envoy\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.666748 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.666729 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-podinfo\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.671858 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.671841 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-token\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.675194 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.675173 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brl8d\" (UniqueName: \"kubernetes.io/projected/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-kube-api-access-brl8d\") pod \"e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.775574 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.775472 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:14.917889 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.917864 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td"] Jul 30 16:01:14.919759 ip-10-0-142-244 kubenswrapper[2579]: W0730 16:01:14.919723 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e3dbcd_ca05_4c95_a4e4_11cabd3f201d.slice/crio-f0435726cfe3b427a2184fb153c9c021666a132d330683b507186b70c22a7309 WatchSource:0}: Error finding container f0435726cfe3b427a2184fb153c9c021666a132d330683b507186b70c22a7309: Status 404 returned error can't find the container with id f0435726cfe3b427a2184fb153c9c021666a132d330683b507186b70c22a7309 Jul 30 16:01:14.922080 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.922047 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 16:01:14.922197 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.922127 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 16:01:14.922197 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:14.922172 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 16:01:15.877196 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:15.877160 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" event={"ID":"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d","Type":"ContainerStarted","Data":"dbf3a76b03976437ce034872fb4ab054e48129633d19480d936a25e3131dbbf7"} Jul 30 16:01:15.877593 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:15.877216 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" event={"ID":"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d","Type":"ContainerStarted","Data":"f0435726cfe3b427a2184fb153c9c021666a132d330683b507186b70c22a7309"} Jul 30 16:01:15.901966 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:15.901919 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" podStartSLOduration=1.901901945 podStartE2EDuration="1.901901945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 16:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 16:01:15.900099482 +0000 UTC m=+3103.135230330" watchObservedRunningTime="2026-07-30 16:01:15.901901945 +0000 UTC m=+3103.137032769" Jul 30 16:01:16.776186 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:16.776149 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:16.780967 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:16.780941 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:16.882400 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:16.882369 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:16.883340 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:16.883318 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:19.725007 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:19.724972 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62"] Jul 30 16:01:19.727858 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:19.727840 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62" Jul 30 16:01:19.731592 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:19.731566 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-5468b9\"" Jul 30 16:01:19.731840 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:19.731823 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-5468b9-dockercfg-x7jwl\"" Jul 30 16:01:19.754342 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:19.754310 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62"] Jul 30 16:01:19.914931 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:19.914889 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade26a0e-afc0-45c9-8ded-8d84da122f2e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62\" (UID: \"ade26a0e-afc0-45c9-8ded-8d84da122f2e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62" Jul 30 16:01:19.915104 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:19.914957 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqsjh\" (UniqueName: \"kubernetes.io/projected/ade26a0e-afc0-45c9-8ded-8d84da122f2e-kube-api-access-sqsjh\") pod \"payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62\" (UID: \"ade26a0e-afc0-45c9-8ded-8d84da122f2e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62" Jul 30 16:01:20.015585 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:20.015495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade26a0e-afc0-45c9-8ded-8d84da122f2e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62\" (UID: \"ade26a0e-afc0-45c9-8ded-8d84da122f2e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62" Jul 30 16:01:20.015585 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:20.015546 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqsjh\" (UniqueName: \"kubernetes.io/projected/ade26a0e-afc0-45c9-8ded-8d84da122f2e-kube-api-access-sqsjh\") pod \"payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62\" (UID: \"ade26a0e-afc0-45c9-8ded-8d84da122f2e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62" Jul 30 16:01:20.016240 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:20.016219 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade26a0e-afc0-45c9-8ded-8d84da122f2e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62\" (UID: \"ade26a0e-afc0-45c9-8ded-8d84da122f2e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62" Jul 30 16:01:20.023685 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:20.023664 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqsjh\" (UniqueName: \"kubernetes.io/projected/ade26a0e-afc0-45c9-8ded-8d84da122f2e-kube-api-access-sqsjh\") pod \"payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62\" (UID: \"ade26a0e-afc0-45c9-8ded-8d84da122f2e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62" Jul 30 16:01:20.038614 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:20.038590 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62" Jul 30 16:01:20.171457 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:20.171428 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62"] Jul 30 16:01:20.173054 ip-10-0-142-244 kubenswrapper[2579]: W0730 16:01:20.173031 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade26a0e_afc0_45c9_8ded_8d84da122f2e.slice/crio-171c9d8c13580b62dec6c58d16093207cbf4642ed3a64f5282e2e1ea01024b1e WatchSource:0}: Error finding container 171c9d8c13580b62dec6c58d16093207cbf4642ed3a64f5282e2e1ea01024b1e: Status 404 returned error can't find the container with id 171c9d8c13580b62dec6c58d16093207cbf4642ed3a64f5282e2e1ea01024b1e Jul 30 16:01:20.904827 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:20.904774 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62" event={"ID":"ade26a0e-afc0-45c9-8ded-8d84da122f2e","Type":"ContainerStarted","Data":"d7b57372d85d25d0aef8544541c42684be5f8050447550464a80df9f32423120"} Jul 30 16:01:20.905300 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:20.904844 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62" event={"ID":"ade26a0e-afc0-45c9-8ded-8d84da122f2e","Type":"ContainerStarted","Data":"171c9d8c13580b62dec6c58d16093207cbf4642ed3a64f5282e2e1ea01024b1e"} Jul 30 16:01:20.905300 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:20.904865 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62" Jul 30 16:01:20.921485 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:20.921428 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62" podStartSLOduration=1.9214140400000002 podStartE2EDuration="1.92141404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 16:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 16:01:20.920195378 +0000 UTC m=+3108.155326225" watchObservedRunningTime="2026-07-30 16:01:20.92141404 +0000 UTC m=+3108.156544885" Jul 30 16:01:25.536335 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:25.536298 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:01:31.911891 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:31.911853 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62" Jul 30 16:01:32.923959 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:32.923917 2579 patch_prober.go:28] interesting pod/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 30 16:01:32.924339 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:32.923969 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" podUID="03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 30 16:01:33.977630 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:33.977602 2579 generic.go:356] "Generic (PLEG): container finished" podID="03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" containerID="dbf3a76b03976437ce034872fb4ab054e48129633d19480d936a25e3131dbbf7" exitCode=0 Jul 30 16:01:33.978038 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:33.977683 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" event={"ID":"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d","Type":"ContainerDied","Data":"dbf3a76b03976437ce034872fb4ab054e48129633d19480d936a25e3131dbbf7"} Jul 30 16:01:33.978102 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:33.978079 2579 scope.go:117] "RemoveContainer" containerID="dbf3a76b03976437ce034872fb4ab054e48129633d19480d936a25e3131dbbf7" Jul 30 16:01:33.979105 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:33.979074 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 16:01:33.979221 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:33.979142 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 16:01:33.979221 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:33.979172 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 16:01:34.984988 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:34.984949 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" event={"ID":"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d","Type":"ContainerStarted","Data":"a38cb72716fa315f6e33e90903861202ba3fea82885e2f1d79cd5f4fade6a4b0"} Jul 30 16:01:35.776254 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:35.776201 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:35.776452 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:35.776431 2579 patch_prober.go:28] interesting pod/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 30 16:01:35.776504 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:35.776475 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" podUID="03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 30 16:01:36.776085 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:36.776045 2579 patch_prober.go:28] interesting pod/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 30 16:01:36.776489 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:36.776104 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" podUID="03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 30 16:01:37.776642 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:37.776599 2579 patch_prober.go:28] interesting pod/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 30 16:01:37.777170 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:37.776651 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" podUID="03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 30 16:01:38.775864 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:38.775820 2579 patch_prober.go:28] interesting pod/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 30 16:01:38.776039 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:38.775877 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" podUID="03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 30 16:01:39.779382 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:39.779349 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:39.779765 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:39.779597 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:39.780339 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:39.780322 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:01:51.657020 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:01:51.656986 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:02:15.237111 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:02:15.237029 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:02:39.283369 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:02:39.283339 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:03:05.232812 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:03:05.232766 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:03:22.642646 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:03:22.642614 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:03:47.733926 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:03:47.733846 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:04:04.652747 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:04:04.652710 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:04:31.933886 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:04:31.933852 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:04:48.630301 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:04:48.630266 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:05:16.733075 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:05:16.732995 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:05:34.629933 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:05:34.629902 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:06:02.230309 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:02.230271 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:06:18.834971 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:18.834936 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:06:44.431057 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:44.431019 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:06:58.344884 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.344852 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98"] Jul 30 16:06:58.348458 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.348441 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.352425 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.352359 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f2cf4d\"/\"kube-root-ca.crt\"" Jul 30 16:06:58.353856 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.353691 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f2cf4d\"/\"openshift-service-ca.crt\"" Jul 30 16:06:58.353856 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.353703 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f2cf4d\"/\"default-dockercfg-lvkhp\"" Jul 30 16:06:58.353856 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.353825 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f2cf4d\"/\"sub-test-model-f2cf4d-kserve-self-signed-certs\"" Jul 30 16:06:58.362834 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.362779 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98"] Jul 30 16:06:58.535049 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.535019 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-tmp-dir\") pod \"sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.535218 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.535068 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksbw6\" (UniqueName: \"kubernetes.io/projected/470626df-6a89-4c24-88e5-a5ff2e78ba7e-kube-api-access-ksbw6\") pod \"sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.535218 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.535103 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-model-cache\") pod \"sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.535218 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.535128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-dshm\") pod \"sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.535218 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.535200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-home\") pod \"sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.535357 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.535220 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/470626df-6a89-4c24-88e5-a5ff2e78ba7e-tls-certs\") pod \"sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.636117 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.636092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksbw6\" (UniqueName: \"kubernetes.io/projected/470626df-6a89-4c24-88e5-a5ff2e78ba7e-kube-api-access-ksbw6\") pod \"sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.636272 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.636130 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-model-cache\") pod \"sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.636272 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.636148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-dshm\") pod \"sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.636272 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.636206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-home\") pod \"sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.636272 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.636225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/470626df-6a89-4c24-88e5-a5ff2e78ba7e-tls-certs\") pod \"sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.636476 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.636287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-tmp-dir\") pod \"sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.636580 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.636556 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-model-cache\") pod \"sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.636678 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.636655 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-home\") pod \"sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.636892 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.636871 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-tmp-dir\") pod \"sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.638455 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.638434 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-dshm\") pod \"sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.638887 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.638870 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/470626df-6a89-4c24-88e5-a5ff2e78ba7e-tls-certs\") pod \"sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.645987 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.645968 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksbw6\" (UniqueName: \"kubernetes.io/projected/470626df-6a89-4c24-88e5-a5ff2e78ba7e-kube-api-access-ksbw6\") pod \"sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.658864 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.658842 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:06:58.788204 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:58.788132 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98"] Jul 30 16:06:58.790597 ip-10-0-142-244 kubenswrapper[2579]: W0730 16:06:58.790568 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod470626df_6a89_4c24_88e5_a5ff2e78ba7e.slice/crio-a542684bd453cbbd73216af23a007ce9510d48702199434e877e138e9dc24238 WatchSource:0}: Error finding container a542684bd453cbbd73216af23a007ce9510d48702199434e877e138e9dc24238: Status 404 returned error can't find the container with id a542684bd453cbbd73216af23a007ce9510d48702199434e877e138e9dc24238 Jul 30 16:06:59.606578 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:59.606538 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" event={"ID":"470626df-6a89-4c24-88e5-a5ff2e78ba7e","Type":"ContainerStarted","Data":"40ecd5645b6312b6d2bea23301e06e10d80df4479f196e3cb2490c758a188ce9"} Jul 30 16:06:59.606966 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:59.606586 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" event={"ID":"470626df-6a89-4c24-88e5-a5ff2e78ba7e","Type":"ContainerStarted","Data":"a542684bd453cbbd73216af23a007ce9510d48702199434e877e138e9dc24238"} Jul 30 16:06:59.622703 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:06:59.622660 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" podStartSLOduration=1.622647598 podStartE2EDuration="1.622647598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 16:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 16:06:59.621655354 +0000 UTC m=+3446.856786200" watchObservedRunningTime="2026-07-30 16:06:59.622647598 +0000 UTC m=+3446.857778443" Jul 30 16:07:01.445625 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:07:01.445588 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:07:08.659972 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:07:08.659937 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:07:08.660357 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:07:08.660012 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:07:08.672596 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:07:08.672574 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:07:09.667810 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:07:09.667779 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:07:40.432518 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:07:40.432481 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:08:30.343158 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:08:30.343126 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:08:50.333314 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:08:50.333273 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:09:12.044540 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:12.044503 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:09:17.914142 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:17.914105 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98"] Jul 30 16:09:31.828383 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:31.828302 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:09:32.932963 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:32.932923 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" podUID="470626df-6a89-4c24-88e5-a5ff2e78ba7e" containerName="main" containerID="cri-o://40ecd5645b6312b6d2bea23301e06e10d80df4479f196e3cb2490c758a188ce9" gracePeriod=45 Jul 30 16:09:33.187053 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.186999 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:09:33.295546 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.295518 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-tmp-dir\") pod \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " Jul 30 16:09:33.295711 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.295566 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksbw6\" (UniqueName: \"kubernetes.io/projected/470626df-6a89-4c24-88e5-a5ff2e78ba7e-kube-api-access-ksbw6\") pod \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " Jul 30 16:09:33.295711 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.295601 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-dshm\") pod \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " Jul 30 16:09:33.295711 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.295627 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-home\") pod \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " Jul 30 16:09:33.295926 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.295714 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/470626df-6a89-4c24-88e5-a5ff2e78ba7e-tls-certs\") pod \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " Jul 30 16:09:33.295926 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.295771 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-model-cache\") pod \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\" (UID: \"470626df-6a89-4c24-88e5-a5ff2e78ba7e\") " Jul 30 16:09:33.295926 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.295822 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "470626df-6a89-4c24-88e5-a5ff2e78ba7e" (UID: "470626df-6a89-4c24-88e5-a5ff2e78ba7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:09:33.296078 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.296011 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-home" (OuterVolumeSpecName: "home") pod "470626df-6a89-4c24-88e5-a5ff2e78ba7e" (UID: "470626df-6a89-4c24-88e5-a5ff2e78ba7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:09:33.296078 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.296019 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "470626df-6a89-4c24-88e5-a5ff2e78ba7e" (UID: "470626df-6a89-4c24-88e5-a5ff2e78ba7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:09:33.296227 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.296208 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-home\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:09:33.296291 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.296232 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-model-cache\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:09:33.296291 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.296244 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-tmp-dir\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:09:33.297826 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.297771 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-dshm" (OuterVolumeSpecName: "dshm") pod "470626df-6a89-4c24-88e5-a5ff2e78ba7e" (UID: "470626df-6a89-4c24-88e5-a5ff2e78ba7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:09:33.297933 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.297888 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/470626df-6a89-4c24-88e5-a5ff2e78ba7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "470626df-6a89-4c24-88e5-a5ff2e78ba7e" (UID: "470626df-6a89-4c24-88e5-a5ff2e78ba7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 16:09:33.297933 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.297899 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470626df-6a89-4c24-88e5-a5ff2e78ba7e-kube-api-access-ksbw6" (OuterVolumeSpecName: "kube-api-access-ksbw6") pod "470626df-6a89-4c24-88e5-a5ff2e78ba7e" (UID: "470626df-6a89-4c24-88e5-a5ff2e78ba7e"). InnerVolumeSpecName "kube-api-access-ksbw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:09:33.382254 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.382228 2579 generic.go:356] "Generic (PLEG): container finished" podID="470626df-6a89-4c24-88e5-a5ff2e78ba7e" containerID="40ecd5645b6312b6d2bea23301e06e10d80df4479f196e3cb2490c758a188ce9" exitCode=0 Jul 30 16:09:33.382384 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.382271 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" Jul 30 16:09:33.382384 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.382311 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" event={"ID":"470626df-6a89-4c24-88e5-a5ff2e78ba7e","Type":"ContainerDied","Data":"40ecd5645b6312b6d2bea23301e06e10d80df4479f196e3cb2490c758a188ce9"} Jul 30 16:09:33.382384 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.382349 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98" event={"ID":"470626df-6a89-4c24-88e5-a5ff2e78ba7e","Type":"ContainerDied","Data":"a542684bd453cbbd73216af23a007ce9510d48702199434e877e138e9dc24238"} Jul 30 16:09:33.382384 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.382367 2579 scope.go:117] "RemoveContainer" containerID="40ecd5645b6312b6d2bea23301e06e10d80df4479f196e3cb2490c758a188ce9" Jul 30 16:09:33.391866 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.391780 2579 scope.go:117] "RemoveContainer" containerID="40ecd5645b6312b6d2bea23301e06e10d80df4479f196e3cb2490c758a188ce9" Jul 30 16:09:33.392060 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:09:33.392041 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ecd5645b6312b6d2bea23301e06e10d80df4479f196e3cb2490c758a188ce9\": container with ID starting with 40ecd5645b6312b6d2bea23301e06e10d80df4479f196e3cb2490c758a188ce9 not found: ID does not exist" containerID="40ecd5645b6312b6d2bea23301e06e10d80df4479f196e3cb2490c758a188ce9" Jul 30 16:09:33.392117 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.392068 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ecd5645b6312b6d2bea23301e06e10d80df4479f196e3cb2490c758a188ce9"} err="failed to get container status \"40ecd5645b6312b6d2bea23301e06e10d80df4479f196e3cb2490c758a188ce9\": rpc error: code = NotFound desc = could not find container \"40ecd5645b6312b6d2bea23301e06e10d80df4479f196e3cb2490c758a188ce9\": container with ID starting with 40ecd5645b6312b6d2bea23301e06e10d80df4479f196e3cb2490c758a188ce9 not found: ID does not exist" Jul 30 16:09:33.396961 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.396943 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/470626df-6a89-4c24-88e5-a5ff2e78ba7e-tls-certs\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:09:33.397047 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.396962 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksbw6\" (UniqueName: \"kubernetes.io/projected/470626df-6a89-4c24-88e5-a5ff2e78ba7e-kube-api-access-ksbw6\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:09:33.397047 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.396972 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/470626df-6a89-4c24-88e5-a5ff2e78ba7e-dshm\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:09:33.405115 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.405094 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98"] Jul 30 16:09:33.409209 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:33.409190 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-f2cf4d/sub-test-model-f2cf4d-kserve-6dc75fc677-hnt98"] Jul 30 16:09:35.198113 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:35.198076 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="470626df-6a89-4c24-88e5-a5ff2e78ba7e" path="/var/lib/kubelet/pods/470626df-6a89-4c24-88e5-a5ff2e78ba7e/volumes" Jul 30 16:09:41.319027 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.318994 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq"] Jul 30 16:09:41.319413 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.319390 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="470626df-6a89-4c24-88e5-a5ff2e78ba7e" containerName="main" Jul 30 16:09:41.319413 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.319401 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="470626df-6a89-4c24-88e5-a5ff2e78ba7e" containerName="main" Jul 30 16:09:41.319490 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.319457 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="470626df-6a89-4c24-88e5-a5ff2e78ba7e" containerName="main" Jul 30 16:09:41.323701 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.323677 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.326491 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.326471 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5468b9\"/\"kube-root-ca.crt\"" Jul 30 16:09:41.327554 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.327521 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5468b9\"/\"rate-test-model-5468b9-kserve-self-signed-certs\"" Jul 30 16:09:41.327959 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.327939 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5468b9\"/\"openshift-service-ca.crt\"" Jul 30 16:09:41.328435 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.328409 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5468b9\"/\"default-dockercfg-swktg\"" Jul 30 16:09:41.332469 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.331935 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq"] Jul 30 16:09:41.368179 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.368156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-model-cache\") pod \"rate-test-model-5468b9-kserve-54dcf887ff-dscpq\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.368289 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.368207 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ce06b5-8100-45c6-9d9c-604ff9e47e74-tls-certs\") pod \"rate-test-model-5468b9-kserve-54dcf887ff-dscpq\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.368335 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.368305 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-dshm\") pod \"rate-test-model-5468b9-kserve-54dcf887ff-dscpq\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.368376 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.368341 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-tmp-dir\") pod \"rate-test-model-5468b9-kserve-54dcf887ff-dscpq\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.368376 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.368361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xxx2\" (UniqueName: \"kubernetes.io/projected/81ce06b5-8100-45c6-9d9c-604ff9e47e74-kube-api-access-7xxx2\") pod \"rate-test-model-5468b9-kserve-54dcf887ff-dscpq\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.368445 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.368379 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-home\") pod \"rate-test-model-5468b9-kserve-54dcf887ff-dscpq\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.469267 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.469238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-model-cache\") pod \"rate-test-model-5468b9-kserve-54dcf887ff-dscpq\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.469406 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.469281 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ce06b5-8100-45c6-9d9c-604ff9e47e74-tls-certs\") pod \"rate-test-model-5468b9-kserve-54dcf887ff-dscpq\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.469406 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.469317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-dshm\") pod \"rate-test-model-5468b9-kserve-54dcf887ff-dscpq\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.469406 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.469340 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-tmp-dir\") pod \"rate-test-model-5468b9-kserve-54dcf887ff-dscpq\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.469406 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.469357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xxx2\" (UniqueName: \"kubernetes.io/projected/81ce06b5-8100-45c6-9d9c-604ff9e47e74-kube-api-access-7xxx2\") pod \"rate-test-model-5468b9-kserve-54dcf887ff-dscpq\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.469607 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.469501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-home\") pod \"rate-test-model-5468b9-kserve-54dcf887ff-dscpq\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.469660 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.469607 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-model-cache\") pod \"rate-test-model-5468b9-kserve-54dcf887ff-dscpq\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.469660 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.469649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-tmp-dir\") pod \"rate-test-model-5468b9-kserve-54dcf887ff-dscpq\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.469893 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.469870 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-home\") pod \"rate-test-model-5468b9-kserve-54dcf887ff-dscpq\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.471527 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.471509 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-dshm\") pod \"rate-test-model-5468b9-kserve-54dcf887ff-dscpq\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.471721 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.471705 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ce06b5-8100-45c6-9d9c-604ff9e47e74-tls-certs\") pod \"rate-test-model-5468b9-kserve-54dcf887ff-dscpq\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.477347 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.477316 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xxx2\" (UniqueName: \"kubernetes.io/projected/81ce06b5-8100-45c6-9d9c-604ff9e47e74-kube-api-access-7xxx2\") pod \"rate-test-model-5468b9-kserve-54dcf887ff-dscpq\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.635602 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.635570 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:41.771238 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:41.771213 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq"] Jul 30 16:09:42.428491 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:42.428455 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" event={"ID":"81ce06b5-8100-45c6-9d9c-604ff9e47e74","Type":"ContainerStarted","Data":"e83ed2c62addb44da6feb2646fdd17e8077fc77c4e4d6e2fee08d31eb64bbc4a"} Jul 30 16:09:42.428939 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:42.428505 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" event={"ID":"81ce06b5-8100-45c6-9d9c-604ff9e47e74","Type":"ContainerStarted","Data":"50d933eb3783b433f48c5477250bbed06db84e9a93f5620c4b1aea6b009228de"} Jul 30 16:09:42.446834 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:42.446774 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" podStartSLOduration=1.446758974 podStartE2EDuration="1.446758974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 16:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 16:09:42.44490219 +0000 UTC m=+3609.680033038" watchObservedRunningTime="2026-07-30 16:09:42.446758974 +0000 UTC m=+3609.681889842" Jul 30 16:09:45.337201 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:45.337163 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:09:51.636722 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:51.636690 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:51.637125 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:51.636738 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:51.649051 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:51.649027 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:09:52.491002 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:09:52.490973 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:10:28.036182 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:10:28.036145 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:10:46.634315 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:10:46.634279 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:11:05.638307 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:05.638225 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:11:11.840197 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:11.840163 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq"] Jul 30 16:11:22.533777 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:22.533739 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:11:26.859983 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:26.859941 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" podUID="81ce06b5-8100-45c6-9d9c-604ff9e47e74" containerName="main" containerID="cri-o://e83ed2c62addb44da6feb2646fdd17e8077fc77c4e4d6e2fee08d31eb64bbc4a" gracePeriod=45 Jul 30 16:11:27.109316 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.109294 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:11:27.167206 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.167179 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-dshm\") pod \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " Jul 30 16:11:27.167345 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.167215 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-model-cache\") pod \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " Jul 30 16:11:27.167345 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.167257 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ce06b5-8100-45c6-9d9c-604ff9e47e74-tls-certs\") pod \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " Jul 30 16:11:27.167345 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.167277 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xxx2\" (UniqueName: \"kubernetes.io/projected/81ce06b5-8100-45c6-9d9c-604ff9e47e74-kube-api-access-7xxx2\") pod \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " Jul 30 16:11:27.167537 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.167448 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-model-cache" (OuterVolumeSpecName: "model-cache") pod "81ce06b5-8100-45c6-9d9c-604ff9e47e74" (UID: "81ce06b5-8100-45c6-9d9c-604ff9e47e74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:11:27.167537 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.167471 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-tmp-dir\") pod \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " Jul 30 16:11:27.167650 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.167558 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-home\") pod \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\" (UID: \"81ce06b5-8100-45c6-9d9c-604ff9e47e74\") " Jul 30 16:11:27.167650 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.167617 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81ce06b5-8100-45c6-9d9c-604ff9e47e74" (UID: "81ce06b5-8100-45c6-9d9c-604ff9e47e74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:11:27.167833 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.167788 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-home" (OuterVolumeSpecName: "home") pod "81ce06b5-8100-45c6-9d9c-604ff9e47e74" (UID: "81ce06b5-8100-45c6-9d9c-604ff9e47e74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:11:27.167948 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.167926 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-tmp-dir\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:11:27.168014 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.167950 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-home\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:11:27.168014 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.167960 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-model-cache\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:11:27.169465 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.169437 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-dshm" (OuterVolumeSpecName: "dshm") pod "81ce06b5-8100-45c6-9d9c-604ff9e47e74" (UID: "81ce06b5-8100-45c6-9d9c-604ff9e47e74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:11:27.169465 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.169447 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81ce06b5-8100-45c6-9d9c-604ff9e47e74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81ce06b5-8100-45c6-9d9c-604ff9e47e74" (UID: "81ce06b5-8100-45c6-9d9c-604ff9e47e74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 16:11:27.169594 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.169503 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ce06b5-8100-45c6-9d9c-604ff9e47e74-kube-api-access-7xxx2" (OuterVolumeSpecName: "kube-api-access-7xxx2") pod "81ce06b5-8100-45c6-9d9c-604ff9e47e74" (UID: "81ce06b5-8100-45c6-9d9c-604ff9e47e74"). InnerVolumeSpecName "kube-api-access-7xxx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:11:27.269057 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.269036 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ce06b5-8100-45c6-9d9c-604ff9e47e74-dshm\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:11:27.269057 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.269058 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ce06b5-8100-45c6-9d9c-604ff9e47e74-tls-certs\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:11:27.269272 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.269080 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xxx2\" (UniqueName: \"kubernetes.io/projected/81ce06b5-8100-45c6-9d9c-604ff9e47e74-kube-api-access-7xxx2\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:11:27.972107 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.972072 2579 generic.go:356] "Generic (PLEG): container finished" podID="81ce06b5-8100-45c6-9d9c-604ff9e47e74" containerID="e83ed2c62addb44da6feb2646fdd17e8077fc77c4e4d6e2fee08d31eb64bbc4a" exitCode=0 Jul 30 16:11:27.972542 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.972123 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" Jul 30 16:11:27.972542 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.972157 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" event={"ID":"81ce06b5-8100-45c6-9d9c-604ff9e47e74","Type":"ContainerDied","Data":"e83ed2c62addb44da6feb2646fdd17e8077fc77c4e4d6e2fee08d31eb64bbc4a"} Jul 30 16:11:27.972542 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.972200 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq" event={"ID":"81ce06b5-8100-45c6-9d9c-604ff9e47e74","Type":"ContainerDied","Data":"50d933eb3783b433f48c5477250bbed06db84e9a93f5620c4b1aea6b009228de"} Jul 30 16:11:27.972542 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.972218 2579 scope.go:117] "RemoveContainer" containerID="e83ed2c62addb44da6feb2646fdd17e8077fc77c4e4d6e2fee08d31eb64bbc4a" Jul 30 16:11:27.981117 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.981097 2579 scope.go:117] "RemoveContainer" containerID="e83ed2c62addb44da6feb2646fdd17e8077fc77c4e4d6e2fee08d31eb64bbc4a" Jul 30 16:11:27.981348 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:11:27.981323 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83ed2c62addb44da6feb2646fdd17e8077fc77c4e4d6e2fee08d31eb64bbc4a\": container with ID starting with e83ed2c62addb44da6feb2646fdd17e8077fc77c4e4d6e2fee08d31eb64bbc4a not found: ID does not exist" containerID="e83ed2c62addb44da6feb2646fdd17e8077fc77c4e4d6e2fee08d31eb64bbc4a" Jul 30 16:11:27.981397 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.981355 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83ed2c62addb44da6feb2646fdd17e8077fc77c4e4d6e2fee08d31eb64bbc4a"} err="failed to get container status \"e83ed2c62addb44da6feb2646fdd17e8077fc77c4e4d6e2fee08d31eb64bbc4a\": rpc error: code = NotFound desc = could not find container \"e83ed2c62addb44da6feb2646fdd17e8077fc77c4e4d6e2fee08d31eb64bbc4a\": container with ID starting with e83ed2c62addb44da6feb2646fdd17e8077fc77c4e4d6e2fee08d31eb64bbc4a not found: ID does not exist" Jul 30 16:11:27.989212 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.989190 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq"] Jul 30 16:11:27.995320 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:27.995300 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5468b9/rate-test-model-5468b9-kserve-54dcf887ff-dscpq"] Jul 30 16:11:29.197379 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:29.197337 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81ce06b5-8100-45c6-9d9c-604ff9e47e74" path="/var/lib/kubelet/pods/81ce06b5-8100-45c6-9d9c-604ff9e47e74/volumes" Jul 30 16:11:31.902963 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:31.902927 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp"] Jul 30 16:11:31.903580 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:31.903559 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81ce06b5-8100-45c6-9d9c-604ff9e47e74" containerName="main" Jul 30 16:11:31.903675 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:31.903582 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="81ce06b5-8100-45c6-9d9c-604ff9e47e74" containerName="main" Jul 30 16:11:31.903732 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:31.903687 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="81ce06b5-8100-45c6-9d9c-604ff9e47e74" containerName="main" Jul 30 16:11:31.908111 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:31.908089 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:31.910778 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:31.910760 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6bc161-openshift-default-dockercfg-xlqxk\"" Jul 30 16:11:31.919271 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:31.919246 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp"] Jul 30 16:11:32.011340 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.011309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-token\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.011502 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.011349 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-credential-socket\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.011502 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.011373 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79d9v\" (UniqueName: \"kubernetes.io/projected/f5e95c65-c797-45c2-83f9-ae3939823ecb-kube-api-access-79d9v\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.011502 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.011425 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-workload-socket\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.011502 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.011480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e95c65-c797-45c2-83f9-ae3939823ecb-istiod-ca-cert\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.011682 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.011512 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-podinfo\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.011682 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.011536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-data\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.011682 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.011566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-envoy\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.011682 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.011589 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-workload-certs\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.112897 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.112854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-token\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.112897 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.112909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-credential-socket\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.113164 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.112937 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79d9v\" (UniqueName: \"kubernetes.io/projected/f5e95c65-c797-45c2-83f9-ae3939823ecb-kube-api-access-79d9v\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.113164 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.112984 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-workload-socket\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.113164 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.113060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e95c65-c797-45c2-83f9-ae3939823ecb-istiod-ca-cert\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.113164 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.113110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-podinfo\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.113164 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.113131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-data\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.113164 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.113153 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-envoy\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.113493 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.113177 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-workload-certs\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.113493 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.113321 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-credential-socket\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.113493 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.113414 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-workload-socket\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.113588 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.113490 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-data\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.113588 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.113521 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-workload-certs\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.113699 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.113680 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e95c65-c797-45c2-83f9-ae3939823ecb-istiod-ca-cert\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.115460 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.115437 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-envoy\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.115599 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.115567 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-podinfo\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.120807 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.120769 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-token\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.121195 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.121173 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79d9v\" (UniqueName: \"kubernetes.io/projected/f5e95c65-c797-45c2-83f9-ae3939823ecb-kube-api-access-79d9v\") pod \"e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.220474 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.220394 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:32.361658 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.361630 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp"] Jul 30 16:11:32.362162 ip-10-0-142-244 kubenswrapper[2579]: W0730 16:11:32.362140 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e95c65_c797_45c2_83f9_ae3939823ecb.slice/crio-5f4d94caa431a1f8db4dcaf95cccaac508ef65692089e46eeadba71a6917e003 WatchSource:0}: Error finding container 5f4d94caa431a1f8db4dcaf95cccaac508ef65692089e46eeadba71a6917e003: Status 404 returned error can't find the container with id 5f4d94caa431a1f8db4dcaf95cccaac508ef65692089e46eeadba71a6917e003 Jul 30 16:11:32.364235 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.364203 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 16:11:32.364318 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.364276 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 16:11:32.364318 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.364304 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 16:11:32.999703 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.999666 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" event={"ID":"f5e95c65-c797-45c2-83f9-ae3939823ecb","Type":"ContainerStarted","Data":"c928e44bbe0144425bd19802b4ea1578ee8221822a46c35a8f4079055a98fc8b"} Jul 30 16:11:33.000096 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:32.999725 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" event={"ID":"f5e95c65-c797-45c2-83f9-ae3939823ecb","Type":"ContainerStarted","Data":"5f4d94caa431a1f8db4dcaf95cccaac508ef65692089e46eeadba71a6917e003"} Jul 30 16:11:33.019237 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:33.019190 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" podStartSLOduration=2.019174636 podStartE2EDuration="2.019174636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 16:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 16:11:33.016980631 +0000 UTC m=+3720.252111481" watchObservedRunningTime="2026-07-30 16:11:33.019174636 +0000 UTC m=+3720.254305483" Jul 30 16:11:33.220864 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:33.220832 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:33.225693 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:33.225669 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:34.004551 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:34.004477 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:34.005298 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:34.005279 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:44.037195 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:44.037158 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:11:53.108420 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:53.108385 2579 generic.go:356] "Generic (PLEG): container finished" podID="f5e95c65-c797-45c2-83f9-ae3939823ecb" containerID="c928e44bbe0144425bd19802b4ea1578ee8221822a46c35a8f4079055a98fc8b" exitCode=0 Jul 30 16:11:53.108852 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:53.108444 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" event={"ID":"f5e95c65-c797-45c2-83f9-ae3939823ecb","Type":"ContainerDied","Data":"c928e44bbe0144425bd19802b4ea1578ee8221822a46c35a8f4079055a98fc8b"} Jul 30 16:11:53.108906 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:53.108893 2579 scope.go:117] "RemoveContainer" containerID="c928e44bbe0144425bd19802b4ea1578ee8221822a46c35a8f4079055a98fc8b" Jul 30 16:11:53.110042 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:53.110010 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 16:11:53.110126 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:53.110080 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 16:11:53.110126 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:53.110110 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 30 16:11:54.115473 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:54.115435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" event={"ID":"f5e95c65-c797-45c2-83f9-ae3939823ecb","Type":"ContainerStarted","Data":"8f0ef38dbe84d0d6877324ef7694f362e95cda64ab0c591cb8881ed37e91fc78"} Jul 30 16:11:54.221119 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:54.221087 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:54.221418 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:54.221395 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 30 16:11:54.221476 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:54.221442 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" podUID="f5e95c65-c797-45c2-83f9-ae3939823ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 30 16:11:55.221083 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:55.221047 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 30 16:11:55.221485 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:55.221097 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" podUID="f5e95c65-c797-45c2-83f9-ae3939823ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 30 16:11:56.220958 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:56.220922 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 30 16:11:56.221137 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:56.220977 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" podUID="f5e95c65-c797-45c2-83f9-ae3939823ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 30 16:11:57.221341 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:57.221309 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 30 16:11:57.221732 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:57.221390 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" podUID="f5e95c65-c797-45c2-83f9-ae3939823ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 30 16:11:58.224145 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:58.224114 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:58.224533 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:58.224442 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:11:58.225098 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:11:58.225081 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:12:09.764685 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:09.764647 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9"] Jul 30 16:12:09.767664 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:09.767646 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9" Jul 30 16:12:09.772193 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:09.772173 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ae9970-dockercfg-d749t\"" Jul 30 16:12:09.772489 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:09.772474 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ae9970\"" Jul 30 16:12:09.779649 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:09.779625 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9"] Jul 30 16:12:09.798086 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:09.798057 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl"] Jul 30 16:12:09.800627 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:09.800610 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl" Jul 30 16:12:09.808958 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:09.808936 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl"] Jul 30 16:12:09.897015 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:09.896984 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b920e676-ebf3-43de-b103-f24702a65710-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9\" (UID: \"b920e676-ebf3-43de-b103-f24702a65710\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9" Jul 30 16:12:09.897184 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:09.897039 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7191c5c3-8883-467d-84cf-78a61c418733-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl\" (UID: \"7191c5c3-8883-467d-84cf-78a61c418733\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl" Jul 30 16:12:09.897184 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:09.897059 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twpqq\" (UniqueName: \"kubernetes.io/projected/7191c5c3-8883-467d-84cf-78a61c418733-kube-api-access-twpqq\") pod \"payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl\" (UID: \"7191c5c3-8883-467d-84cf-78a61c418733\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl" Jul 30 16:12:09.897184 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:09.897088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxfg5\" (UniqueName: \"kubernetes.io/projected/b920e676-ebf3-43de-b103-f24702a65710-kube-api-access-qxfg5\") pod \"payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9\" (UID: \"b920e676-ebf3-43de-b103-f24702a65710\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9" Jul 30 16:12:09.997935 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:09.997903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7191c5c3-8883-467d-84cf-78a61c418733-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl\" (UID: \"7191c5c3-8883-467d-84cf-78a61c418733\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl" Jul 30 16:12:09.997935 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:09.997936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twpqq\" (UniqueName: \"kubernetes.io/projected/7191c5c3-8883-467d-84cf-78a61c418733-kube-api-access-twpqq\") pod \"payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl\" (UID: \"7191c5c3-8883-467d-84cf-78a61c418733\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl" Jul 30 16:12:09.998146 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:09.998058 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxfg5\" (UniqueName: \"kubernetes.io/projected/b920e676-ebf3-43de-b103-f24702a65710-kube-api-access-qxfg5\") pod \"payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9\" (UID: \"b920e676-ebf3-43de-b103-f24702a65710\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9" Jul 30 16:12:09.998225 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:09.998207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b920e676-ebf3-43de-b103-f24702a65710-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9\" (UID: \"b920e676-ebf3-43de-b103-f24702a65710\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9" Jul 30 16:12:09.998664 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:09.998644 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7191c5c3-8883-467d-84cf-78a61c418733-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl\" (UID: \"7191c5c3-8883-467d-84cf-78a61c418733\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl" Jul 30 16:12:09.998991 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:09.998975 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b920e676-ebf3-43de-b103-f24702a65710-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9\" (UID: \"b920e676-ebf3-43de-b103-f24702a65710\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9" Jul 30 16:12:10.006165 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:10.006142 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxfg5\" (UniqueName: \"kubernetes.io/projected/b920e676-ebf3-43de-b103-f24702a65710-kube-api-access-qxfg5\") pod \"payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9\" (UID: \"b920e676-ebf3-43de-b103-f24702a65710\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9" Jul 30 16:12:10.006267 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:10.006191 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twpqq\" (UniqueName: \"kubernetes.io/projected/7191c5c3-8883-467d-84cf-78a61c418733-kube-api-access-twpqq\") pod \"payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl\" (UID: \"7191c5c3-8883-467d-84cf-78a61c418733\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl" Jul 30 16:12:10.078939 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:10.078860 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9" Jul 30 16:12:10.111458 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:10.111423 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl" Jul 30 16:12:10.222388 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:10.222363 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9"] Jul 30 16:12:10.224652 ip-10-0-142-244 kubenswrapper[2579]: W0730 16:12:10.224623 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb920e676_ebf3_43de_b103_f24702a65710.slice/crio-e3f35ef1a56af0c30b986f707a0b6294fc2b66c09340a1eaf22a0a3dce1313f5 WatchSource:0}: Error finding container e3f35ef1a56af0c30b986f707a0b6294fc2b66c09340a1eaf22a0a3dce1313f5: Status 404 returned error can't find the container with id e3f35ef1a56af0c30b986f707a0b6294fc2b66c09340a1eaf22a0a3dce1313f5 Jul 30 16:12:10.467882 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:10.467854 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl"] Jul 30 16:12:10.469435 ip-10-0-142-244 kubenswrapper[2579]: W0730 16:12:10.469407 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7191c5c3_8883_467d_84cf_78a61c418733.slice/crio-b3272070bb7057fc258a972d598f7560127ad6660276002313ffe92d33d02502 WatchSource:0}: Error finding container b3272070bb7057fc258a972d598f7560127ad6660276002313ffe92d33d02502: Status 404 returned error can't find the container with id b3272070bb7057fc258a972d598f7560127ad6660276002313ffe92d33d02502 Jul 30 16:12:11.214396 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:11.214361 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl" event={"ID":"7191c5c3-8883-467d-84cf-78a61c418733","Type":"ContainerStarted","Data":"883948b109a097e331e9bf90e572b682abb93af889aa18485e634a5b4a440718"} Jul 30 16:12:11.214860 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:11.214419 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl" event={"ID":"7191c5c3-8883-467d-84cf-78a61c418733","Type":"ContainerStarted","Data":"b3272070bb7057fc258a972d598f7560127ad6660276002313ffe92d33d02502"} Jul 30 16:12:11.214860 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:11.214450 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl" Jul 30 16:12:11.215953 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:11.215926 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9" event={"ID":"b920e676-ebf3-43de-b103-f24702a65710","Type":"ContainerStarted","Data":"d3bf73e29274b859a713e885d2d50a2c51924c26ca01c1e2fca42953d206f302"} Jul 30 16:12:11.216092 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:11.215967 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9" event={"ID":"b920e676-ebf3-43de-b103-f24702a65710","Type":"ContainerStarted","Data":"e3f35ef1a56af0c30b986f707a0b6294fc2b66c09340a1eaf22a0a3dce1313f5"} Jul 30 16:12:11.216092 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:11.216046 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9" Jul 30 16:12:11.232406 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:11.232363 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl" podStartSLOduration=2.232349741 podStartE2EDuration="2.232349741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 16:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 16:12:11.230153699 +0000 UTC m=+3758.465284549" watchObservedRunningTime="2026-07-30 16:12:11.232349741 +0000 UTC m=+3758.467480565" Jul 30 16:12:11.246825 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:11.246747 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9" podStartSLOduration=2.246734178 podStartE2EDuration="2.246734178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 16:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 16:12:11.244240654 +0000 UTC m=+3758.479371494" watchObservedRunningTime="2026-07-30 16:12:11.246734178 +0000 UTC m=+3758.481865026" Jul 30 16:12:14.335511 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:14.334717 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:12:22.223749 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:22.223705 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9" Jul 30 16:12:22.224190 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:22.223840 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl" Jul 30 16:12:40.436063 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:40.435983 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl_7191c5c3-8883-467d-84cf-78a61c418733/payload-processing/0.log" Jul 30 16:12:46.444857 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:12:46.444821 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:13:20.438571 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:20.438534 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:13:30.849667 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:30.849634 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl_7191c5c3-8883-467d-84cf-78a61c418733/payload-processing/0.log" Jul 30 16:13:36.436300 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.436265 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb"] Jul 30 16:13:36.440384 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.440363 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb" Jul 30 16:13:36.443228 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.443200 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-812b03\"" Jul 30 16:13:36.443332 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.443255 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-812b03-dockercfg-hqzbf\"" Jul 30 16:13:36.450350 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.450323 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb"] Jul 30 16:13:36.472232 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.472210 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6"] Jul 30 16:13:36.475622 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.475606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6" Jul 30 16:13:36.484227 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.484202 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6"] Jul 30 16:13:36.631731 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.631700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f51b26-8629-445f-aed8-4b3c208f46cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6\" (UID: \"a4f51b26-8629-445f-aed8-4b3c208f46cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6" Jul 30 16:13:36.631939 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.631749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th7j5\" (UniqueName: \"kubernetes.io/projected/a4f51b26-8629-445f-aed8-4b3c208f46cd-kube-api-access-th7j5\") pod \"payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6\" (UID: \"a4f51b26-8629-445f-aed8-4b3c208f46cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6" Jul 30 16:13:36.631939 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.631867 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739747ed-95bb-4f2d-8b7b-8c0fb5025c65-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb\" (UID: \"739747ed-95bb-4f2d-8b7b-8c0fb5025c65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb" Jul 30 16:13:36.631939 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.631898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blzq9\" (UniqueName: \"kubernetes.io/projected/739747ed-95bb-4f2d-8b7b-8c0fb5025c65-kube-api-access-blzq9\") pod \"payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb\" (UID: \"739747ed-95bb-4f2d-8b7b-8c0fb5025c65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb" Jul 30 16:13:36.732640 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.732568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f51b26-8629-445f-aed8-4b3c208f46cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6\" (UID: \"a4f51b26-8629-445f-aed8-4b3c208f46cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6" Jul 30 16:13:36.732640 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.732613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th7j5\" (UniqueName: \"kubernetes.io/projected/a4f51b26-8629-445f-aed8-4b3c208f46cd-kube-api-access-th7j5\") pod \"payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6\" (UID: \"a4f51b26-8629-445f-aed8-4b3c208f46cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6" Jul 30 16:13:36.732640 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.732652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739747ed-95bb-4f2d-8b7b-8c0fb5025c65-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb\" (UID: \"739747ed-95bb-4f2d-8b7b-8c0fb5025c65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb" Jul 30 16:13:36.732640 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.732670 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blzq9\" (UniqueName: \"kubernetes.io/projected/739747ed-95bb-4f2d-8b7b-8c0fb5025c65-kube-api-access-blzq9\") pod \"payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb\" (UID: \"739747ed-95bb-4f2d-8b7b-8c0fb5025c65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb" Jul 30 16:13:36.733420 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.733398 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f51b26-8629-445f-aed8-4b3c208f46cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6\" (UID: \"a4f51b26-8629-445f-aed8-4b3c208f46cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6" Jul 30 16:13:36.733503 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.733437 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739747ed-95bb-4f2d-8b7b-8c0fb5025c65-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb\" (UID: \"739747ed-95bb-4f2d-8b7b-8c0fb5025c65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb" Jul 30 16:13:36.740707 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.740679 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th7j5\" (UniqueName: \"kubernetes.io/projected/a4f51b26-8629-445f-aed8-4b3c208f46cd-kube-api-access-th7j5\") pod \"payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6\" (UID: \"a4f51b26-8629-445f-aed8-4b3c208f46cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6" Jul 30 16:13:36.741009 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.740989 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blzq9\" (UniqueName: \"kubernetes.io/projected/739747ed-95bb-4f2d-8b7b-8c0fb5025c65-kube-api-access-blzq9\") pod \"payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb\" (UID: \"739747ed-95bb-4f2d-8b7b-8c0fb5025c65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb" Jul 30 16:13:36.751750 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.751730 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb" Jul 30 16:13:36.787339 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.787316 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6" Jul 30 16:13:36.897504 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.897400 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb"] Jul 30 16:13:36.901238 ip-10-0-142-244 kubenswrapper[2579]: W0730 16:13:36.901205 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739747ed_95bb_4f2d_8b7b_8c0fb5025c65.slice/crio-473ef0d0857b25c93bbcf7032ead805fed0dd16ffc83248ac4f3659ca7ef339e WatchSource:0}: Error finding container 473ef0d0857b25c93bbcf7032ead805fed0dd16ffc83248ac4f3659ca7ef339e: Status 404 returned error can't find the container with id 473ef0d0857b25c93bbcf7032ead805fed0dd16ffc83248ac4f3659ca7ef339e Jul 30 16:13:36.939976 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:36.939954 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6"] Jul 30 16:13:36.942975 ip-10-0-142-244 kubenswrapper[2579]: W0730 16:13:36.942942 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f51b26_8629_445f_aed8_4b3c208f46cd.slice/crio-57143a5b10de7182c804c3c42382a1641e31c5e38cc6212f751d3414382692c0 WatchSource:0}: Error finding container 57143a5b10de7182c804c3c42382a1641e31c5e38cc6212f751d3414382692c0: Status 404 returned error can't find the container with id 57143a5b10de7182c804c3c42382a1641e31c5e38cc6212f751d3414382692c0 Jul 30 16:13:37.662092 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:37.662052 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6" event={"ID":"a4f51b26-8629-445f-aed8-4b3c208f46cd","Type":"ContainerStarted","Data":"5406c4a30a0a296ed422861e8ec84ddc9eefae141faa40cd03b7d4aafaf31a76"} Jul 30 16:13:37.662861 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:37.662117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6" event={"ID":"a4f51b26-8629-445f-aed8-4b3c208f46cd","Type":"ContainerStarted","Data":"57143a5b10de7182c804c3c42382a1641e31c5e38cc6212f751d3414382692c0"} Jul 30 16:13:37.662861 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:37.662784 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6" Jul 30 16:13:37.663658 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:37.663636 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb" event={"ID":"739747ed-95bb-4f2d-8b7b-8c0fb5025c65","Type":"ContainerStarted","Data":"de1b5cc705fb8f7887ac0a1fcbb08a543064623bbd3af418ed92c396b02058a6"} Jul 30 16:13:37.663792 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:37.663679 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb" event={"ID":"739747ed-95bb-4f2d-8b7b-8c0fb5025c65","Type":"ContainerStarted","Data":"473ef0d0857b25c93bbcf7032ead805fed0dd16ffc83248ac4f3659ca7ef339e"} Jul 30 16:13:37.663792 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:37.663710 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb" Jul 30 16:13:37.678482 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:37.678439 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6" podStartSLOduration=1.678425941 podStartE2EDuration="1.678425941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 16:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 16:13:37.677030471 +0000 UTC m=+3844.912161318" watchObservedRunningTime="2026-07-30 16:13:37.678425941 +0000 UTC m=+3844.913556786" Jul 30 16:13:37.693627 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:37.693585 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb" podStartSLOduration=1.693567493 podStartE2EDuration="1.693567493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 16:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 16:13:37.692048292 +0000 UTC m=+3844.927179161" watchObservedRunningTime="2026-07-30 16:13:37.693567493 +0000 UTC m=+3844.928698343" Jul 30 16:13:40.031590 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:40.031550 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:13:48.671945 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:48.671911 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb" Jul 30 16:13:49.677510 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:13:49.677481 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6" Jul 30 16:14:06.414263 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.414182 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb"] Jul 30 16:14:06.415027 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.414440 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb" podUID="739747ed-95bb-4f2d-8b7b-8c0fb5025c65" containerName="payload-pre-processing" containerID="cri-o://de1b5cc705fb8f7887ac0a1fcbb08a543064623bbd3af418ed92c396b02058a6" gracePeriod=30 Jul 30 16:14:06.417791 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.417765 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6"] Jul 30 16:14:06.418022 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.417972 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6" podUID="a4f51b26-8629-445f-aed8-4b3c208f46cd" containerName="payload-processing" containerID="cri-o://5406c4a30a0a296ed422861e8ec84ddc9eefae141faa40cd03b7d4aafaf31a76" gracePeriod=30 Jul 30 16:14:06.719603 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.719578 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb" Jul 30 16:14:06.722827 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.722783 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6" Jul 30 16:14:06.816536 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.816507 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th7j5\" (UniqueName: \"kubernetes.io/projected/a4f51b26-8629-445f-aed8-4b3c208f46cd-kube-api-access-th7j5\") pod \"a4f51b26-8629-445f-aed8-4b3c208f46cd\" (UID: \"a4f51b26-8629-445f-aed8-4b3c208f46cd\") " Jul 30 16:14:06.816683 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.816546 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739747ed-95bb-4f2d-8b7b-8c0fb5025c65-plugins-config-volume\") pod \"739747ed-95bb-4f2d-8b7b-8c0fb5025c65\" (UID: \"739747ed-95bb-4f2d-8b7b-8c0fb5025c65\") " Jul 30 16:14:06.816683 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.816669 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f51b26-8629-445f-aed8-4b3c208f46cd-plugins-config-volume\") pod \"a4f51b26-8629-445f-aed8-4b3c208f46cd\" (UID: \"a4f51b26-8629-445f-aed8-4b3c208f46cd\") " Jul 30 16:14:06.816901 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.816698 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blzq9\" (UniqueName: \"kubernetes.io/projected/739747ed-95bb-4f2d-8b7b-8c0fb5025c65-kube-api-access-blzq9\") pod \"739747ed-95bb-4f2d-8b7b-8c0fb5025c65\" (UID: \"739747ed-95bb-4f2d-8b7b-8c0fb5025c65\") " Jul 30 16:14:06.817023 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.816998 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f51b26-8629-445f-aed8-4b3c208f46cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a4f51b26-8629-445f-aed8-4b3c208f46cd" (UID: "a4f51b26-8629-445f-aed8-4b3c208f46cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 16:14:06.817088 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.817034 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739747ed-95bb-4f2d-8b7b-8c0fb5025c65-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "739747ed-95bb-4f2d-8b7b-8c0fb5025c65" (UID: "739747ed-95bb-4f2d-8b7b-8c0fb5025c65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 16:14:06.818963 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.818937 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f51b26-8629-445f-aed8-4b3c208f46cd-kube-api-access-th7j5" (OuterVolumeSpecName: "kube-api-access-th7j5") pod "a4f51b26-8629-445f-aed8-4b3c208f46cd" (UID: "a4f51b26-8629-445f-aed8-4b3c208f46cd"). InnerVolumeSpecName "kube-api-access-th7j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:14:06.819079 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.818988 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739747ed-95bb-4f2d-8b7b-8c0fb5025c65-kube-api-access-blzq9" (OuterVolumeSpecName: "kube-api-access-blzq9") pod "739747ed-95bb-4f2d-8b7b-8c0fb5025c65" (UID: "739747ed-95bb-4f2d-8b7b-8c0fb5025c65"). InnerVolumeSpecName "kube-api-access-blzq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:14:06.819287 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.819264 2579 generic.go:356] "Generic (PLEG): container finished" podID="a4f51b26-8629-445f-aed8-4b3c208f46cd" containerID="5406c4a30a0a296ed422861e8ec84ddc9eefae141faa40cd03b7d4aafaf31a76" exitCode=0 Jul 30 16:14:06.819373 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.819311 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6" Jul 30 16:14:06.819373 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.819346 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6" event={"ID":"a4f51b26-8629-445f-aed8-4b3c208f46cd","Type":"ContainerDied","Data":"5406c4a30a0a296ed422861e8ec84ddc9eefae141faa40cd03b7d4aafaf31a76"} Jul 30 16:14:06.819460 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.819424 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6" event={"ID":"a4f51b26-8629-445f-aed8-4b3c208f46cd","Type":"ContainerDied","Data":"57143a5b10de7182c804c3c42382a1641e31c5e38cc6212f751d3414382692c0"} Jul 30 16:14:06.819460 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.819453 2579 scope.go:117] "RemoveContainer" containerID="5406c4a30a0a296ed422861e8ec84ddc9eefae141faa40cd03b7d4aafaf31a76" Jul 30 16:14:06.820762 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.820734 2579 generic.go:356] "Generic (PLEG): container finished" podID="739747ed-95bb-4f2d-8b7b-8c0fb5025c65" containerID="de1b5cc705fb8f7887ac0a1fcbb08a543064623bbd3af418ed92c396b02058a6" exitCode=0 Jul 30 16:14:06.820909 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.820768 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb" event={"ID":"739747ed-95bb-4f2d-8b7b-8c0fb5025c65","Type":"ContainerDied","Data":"de1b5cc705fb8f7887ac0a1fcbb08a543064623bbd3af418ed92c396b02058a6"} Jul 30 16:14:06.820909 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.820781 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb" Jul 30 16:14:06.820909 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.820839 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb" event={"ID":"739747ed-95bb-4f2d-8b7b-8c0fb5025c65","Type":"ContainerDied","Data":"473ef0d0857b25c93bbcf7032ead805fed0dd16ffc83248ac4f3659ca7ef339e"} Jul 30 16:14:06.833174 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.833153 2579 scope.go:117] "RemoveContainer" containerID="5406c4a30a0a296ed422861e8ec84ddc9eefae141faa40cd03b7d4aafaf31a76" Jul 30 16:14:06.833412 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:14:06.833394 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5406c4a30a0a296ed422861e8ec84ddc9eefae141faa40cd03b7d4aafaf31a76\": container with ID starting with 5406c4a30a0a296ed422861e8ec84ddc9eefae141faa40cd03b7d4aafaf31a76 not found: ID does not exist" containerID="5406c4a30a0a296ed422861e8ec84ddc9eefae141faa40cd03b7d4aafaf31a76" Jul 30 16:14:06.833462 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.833421 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5406c4a30a0a296ed422861e8ec84ddc9eefae141faa40cd03b7d4aafaf31a76"} err="failed to get container status \"5406c4a30a0a296ed422861e8ec84ddc9eefae141faa40cd03b7d4aafaf31a76\": rpc error: code = NotFound desc = could not find container \"5406c4a30a0a296ed422861e8ec84ddc9eefae141faa40cd03b7d4aafaf31a76\": container with ID starting with 5406c4a30a0a296ed422861e8ec84ddc9eefae141faa40cd03b7d4aafaf31a76 not found: ID does not exist" Jul 30 16:14:06.833462 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.833439 2579 scope.go:117] "RemoveContainer" containerID="de1b5cc705fb8f7887ac0a1fcbb08a543064623bbd3af418ed92c396b02058a6" Jul 30 16:14:06.842569 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.842553 2579 scope.go:117] "RemoveContainer" containerID="de1b5cc705fb8f7887ac0a1fcbb08a543064623bbd3af418ed92c396b02058a6" Jul 30 16:14:06.842867 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:14:06.842788 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1b5cc705fb8f7887ac0a1fcbb08a543064623bbd3af418ed92c396b02058a6\": container with ID starting with de1b5cc705fb8f7887ac0a1fcbb08a543064623bbd3af418ed92c396b02058a6 not found: ID does not exist" containerID="de1b5cc705fb8f7887ac0a1fcbb08a543064623bbd3af418ed92c396b02058a6" Jul 30 16:14:06.842922 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.842879 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1b5cc705fb8f7887ac0a1fcbb08a543064623bbd3af418ed92c396b02058a6"} err="failed to get container status \"de1b5cc705fb8f7887ac0a1fcbb08a543064623bbd3af418ed92c396b02058a6\": rpc error: code = NotFound desc = could not find container \"de1b5cc705fb8f7887ac0a1fcbb08a543064623bbd3af418ed92c396b02058a6\": container with ID starting with de1b5cc705fb8f7887ac0a1fcbb08a543064623bbd3af418ed92c396b02058a6 not found: ID does not exist" Jul 30 16:14:06.851141 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.851119 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6"] Jul 30 16:14:06.860611 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.860579 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-812b03-8566fcbcbd-d6wl6"] Jul 30 16:14:06.873974 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.873954 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb"] Jul 30 16:14:06.881774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.881751 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-812b03-86dddc6d44-dtnkb"] Jul 30 16:14:06.918259 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.918234 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f51b26-8629-445f-aed8-4b3c208f46cd-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:14:06.918259 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.918257 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blzq9\" (UniqueName: \"kubernetes.io/projected/739747ed-95bb-4f2d-8b7b-8c0fb5025c65-kube-api-access-blzq9\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:14:06.918380 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.918269 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th7j5\" (UniqueName: \"kubernetes.io/projected/a4f51b26-8629-445f-aed8-4b3c208f46cd-kube-api-access-th7j5\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:14:06.918380 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:06.918278 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739747ed-95bb-4f2d-8b7b-8c0fb5025c65-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:14:07.198085 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:07.198057 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="739747ed-95bb-4f2d-8b7b-8c0fb5025c65" path="/var/lib/kubelet/pods/739747ed-95bb-4f2d-8b7b-8c0fb5025c65/volumes" Jul 30 16:14:07.198415 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:07.198402 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4f51b26-8629-445f-aed8-4b3c208f46cd" path="/var/lib/kubelet/pods/a4f51b26-8629-445f-aed8-4b3c208f46cd/volumes" Jul 30 16:14:10.933391 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:10.933354 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:14:31.240617 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:31.240579 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:14:42.038911 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:14:42.038871 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:15:21.525024 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:21.524988 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp"] Jul 30 16:15:21.525631 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:21.525294 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" podUID="f5e95c65-c797-45c2-83f9-ae3939823ecb" containerName="istio-proxy" containerID="cri-o://8f0ef38dbe84d0d6877324ef7694f362e95cda64ab0c591cb8881ed37e91fc78" gracePeriod=30 Jul 30 16:15:26.788385 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.788327 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:15:26.970191 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.970164 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e95c65-c797-45c2-83f9-ae3939823ecb-istiod-ca-cert\") pod \"f5e95c65-c797-45c2-83f9-ae3939823ecb\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " Jul 30 16:15:26.970356 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.970198 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-envoy\") pod \"f5e95c65-c797-45c2-83f9-ae3939823ecb\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " Jul 30 16:15:26.970356 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.970222 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-credential-socket\") pod \"f5e95c65-c797-45c2-83f9-ae3939823ecb\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " Jul 30 16:15:26.970356 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.970239 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79d9v\" (UniqueName: \"kubernetes.io/projected/f5e95c65-c797-45c2-83f9-ae3939823ecb-kube-api-access-79d9v\") pod \"f5e95c65-c797-45c2-83f9-ae3939823ecb\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " Jul 30 16:15:26.970356 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.970309 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-podinfo\") pod \"f5e95c65-c797-45c2-83f9-ae3939823ecb\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " Jul 30 16:15:26.970356 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.970344 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-token\") pod \"f5e95c65-c797-45c2-83f9-ae3939823ecb\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " Jul 30 16:15:26.970635 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.970455 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-workload-socket\") pod \"f5e95c65-c797-45c2-83f9-ae3939823ecb\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " Jul 30 16:15:26.970635 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.970492 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-data\") pod \"f5e95c65-c797-45c2-83f9-ae3939823ecb\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " Jul 30 16:15:26.970635 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.970507 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-workload-certs\") pod \"f5e95c65-c797-45c2-83f9-ae3939823ecb\" (UID: \"f5e95c65-c797-45c2-83f9-ae3939823ecb\") " Jul 30 16:15:26.970635 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.970527 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f5e95c65-c797-45c2-83f9-ae3939823ecb" (UID: "f5e95c65-c797-45c2-83f9-ae3939823ecb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:15:26.970635 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.970572 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e95c65-c797-45c2-83f9-ae3939823ecb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f5e95c65-c797-45c2-83f9-ae3939823ecb" (UID: "f5e95c65-c797-45c2-83f9-ae3939823ecb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 16:15:26.970933 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.970750 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f5e95c65-c797-45c2-83f9-ae3939823ecb" (UID: "f5e95c65-c797-45c2-83f9-ae3939823ecb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:15:26.970933 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.970870 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-workload-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:15:26.970933 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.970876 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f5e95c65-c797-45c2-83f9-ae3939823ecb" (UID: "f5e95c65-c797-45c2-83f9-ae3939823ecb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:15:26.970933 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.970886 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e95c65-c797-45c2-83f9-ae3939823ecb-istiod-ca-cert\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:15:26.970933 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.970924 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-credential-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:15:26.971570 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.971546 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-data" (OuterVolumeSpecName: "istio-data") pod "f5e95c65-c797-45c2-83f9-ae3939823ecb" (UID: "f5e95c65-c797-45c2-83f9-ae3939823ecb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:15:26.972677 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.972660 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e95c65-c797-45c2-83f9-ae3939823ecb-kube-api-access-79d9v" (OuterVolumeSpecName: "kube-api-access-79d9v") pod "f5e95c65-c797-45c2-83f9-ae3939823ecb" (UID: "f5e95c65-c797-45c2-83f9-ae3939823ecb"). InnerVolumeSpecName "kube-api-access-79d9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:15:26.972739 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.972660 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f5e95c65-c797-45c2-83f9-ae3939823ecb" (UID: "f5e95c65-c797-45c2-83f9-ae3939823ecb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 16:15:26.972954 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.972931 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f5e95c65-c797-45c2-83f9-ae3939823ecb" (UID: "f5e95c65-c797-45c2-83f9-ae3939823ecb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:15:26.973054 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:26.972954 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-token" (OuterVolumeSpecName: "istio-token") pod "f5e95c65-c797-45c2-83f9-ae3939823ecb" (UID: "f5e95c65-c797-45c2-83f9-ae3939823ecb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:15:27.072304 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:27.072245 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-data\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:15:27.072304 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:27.072267 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-workload-certs\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:15:27.072304 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:27.072276 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-envoy\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:15:27.072304 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:27.072284 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79d9v\" (UniqueName: \"kubernetes.io/projected/f5e95c65-c797-45c2-83f9-ae3939823ecb-kube-api-access-79d9v\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:15:27.072304 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:27.072293 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-podinfo\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:15:27.072304 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:27.072302 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e95c65-c797-45c2-83f9-ae3939823ecb-istio-token\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:15:27.245229 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:27.245197 2579 generic.go:356] "Generic (PLEG): container finished" podID="f5e95c65-c797-45c2-83f9-ae3939823ecb" containerID="8f0ef38dbe84d0d6877324ef7694f362e95cda64ab0c591cb8881ed37e91fc78" exitCode=0 Jul 30 16:15:27.245343 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:27.245241 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" Jul 30 16:15:27.245343 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:27.245284 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" event={"ID":"f5e95c65-c797-45c2-83f9-ae3939823ecb","Type":"ContainerDied","Data":"8f0ef38dbe84d0d6877324ef7694f362e95cda64ab0c591cb8881ed37e91fc78"} Jul 30 16:15:27.245343 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:27.245338 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp" event={"ID":"f5e95c65-c797-45c2-83f9-ae3939823ecb","Type":"ContainerDied","Data":"5f4d94caa431a1f8db4dcaf95cccaac508ef65692089e46eeadba71a6917e003"} Jul 30 16:15:27.245462 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:27.245361 2579 scope.go:117] "RemoveContainer" containerID="8f0ef38dbe84d0d6877324ef7694f362e95cda64ab0c591cb8881ed37e91fc78" Jul 30 16:15:27.254697 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:27.254682 2579 scope.go:117] "RemoveContainer" containerID="c928e44bbe0144425bd19802b4ea1578ee8221822a46c35a8f4079055a98fc8b" Jul 30 16:15:27.265104 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:27.265087 2579 scope.go:117] "RemoveContainer" containerID="8f0ef38dbe84d0d6877324ef7694f362e95cda64ab0c591cb8881ed37e91fc78" Jul 30 16:15:27.265413 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:15:27.265380 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0ef38dbe84d0d6877324ef7694f362e95cda64ab0c591cb8881ed37e91fc78\": container with ID starting with 8f0ef38dbe84d0d6877324ef7694f362e95cda64ab0c591cb8881ed37e91fc78 not found: ID does not exist" containerID="8f0ef38dbe84d0d6877324ef7694f362e95cda64ab0c591cb8881ed37e91fc78" Jul 30 16:15:27.265471 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:27.265423 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0ef38dbe84d0d6877324ef7694f362e95cda64ab0c591cb8881ed37e91fc78"} err="failed to get container status \"8f0ef38dbe84d0d6877324ef7694f362e95cda64ab0c591cb8881ed37e91fc78\": rpc error: code = NotFound desc = could not find container \"8f0ef38dbe84d0d6877324ef7694f362e95cda64ab0c591cb8881ed37e91fc78\": container with ID starting with 8f0ef38dbe84d0d6877324ef7694f362e95cda64ab0c591cb8881ed37e91fc78 not found: ID does not exist" Jul 30 16:15:27.265471 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:27.265445 2579 scope.go:117] "RemoveContainer" containerID="c928e44bbe0144425bd19802b4ea1578ee8221822a46c35a8f4079055a98fc8b" Jul 30 16:15:27.265657 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:15:27.265638 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c928e44bbe0144425bd19802b4ea1578ee8221822a46c35a8f4079055a98fc8b\": container with ID starting with c928e44bbe0144425bd19802b4ea1578ee8221822a46c35a8f4079055a98fc8b not found: ID does not exist" containerID="c928e44bbe0144425bd19802b4ea1578ee8221822a46c35a8f4079055a98fc8b" Jul 30 16:15:27.265720 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:27.265669 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c928e44bbe0144425bd19802b4ea1578ee8221822a46c35a8f4079055a98fc8b"} err="failed to get container status \"c928e44bbe0144425bd19802b4ea1578ee8221822a46c35a8f4079055a98fc8b\": rpc error: code = NotFound desc = could not find container \"c928e44bbe0144425bd19802b4ea1578ee8221822a46c35a8f4079055a98fc8b\": container with ID starting with c928e44bbe0144425bd19802b4ea1578ee8221822a46c35a8f4079055a98fc8b not found: ID does not exist" Jul 30 16:15:27.265780 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:27.265764 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp"] Jul 30 16:15:27.270962 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:27.270943 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6bc161-openshift-default-c785694b8-vd8lp"] Jul 30 16:15:29.198207 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:29.198171 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5e95c65-c797-45c2-83f9-ae3939823ecb" path="/var/lib/kubelet/pods/f5e95c65-c797-45c2-83f9-ae3939823ecb/volumes" Jul 30 16:15:32.732404 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:32.732367 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl"] Jul 30 16:15:32.732900 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:32.732580 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl" podUID="7191c5c3-8883-467d-84cf-78a61c418733" containerName="payload-processing" containerID="cri-o://883948b109a097e331e9bf90e572b682abb93af889aa18485e634a5b4a440718" gracePeriod=30 Jul 30 16:15:32.743135 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:32.743106 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9"] Jul 30 16:15:32.743357 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:32.743325 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9" podUID="b920e676-ebf3-43de-b103-f24702a65710" containerName="payload-pre-processing" containerID="cri-o://d3bf73e29274b859a713e885d2d50a2c51924c26ca01c1e2fca42953d206f302" gracePeriod=30 Jul 30 16:15:33.040926 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.040899 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl" Jul 30 16:15:33.043618 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.043598 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9" Jul 30 16:15:33.228133 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.228106 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b920e676-ebf3-43de-b103-f24702a65710-plugins-config-volume\") pod \"b920e676-ebf3-43de-b103-f24702a65710\" (UID: \"b920e676-ebf3-43de-b103-f24702a65710\") " Jul 30 16:15:33.228303 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.228197 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7191c5c3-8883-467d-84cf-78a61c418733-plugins-config-volume\") pod \"7191c5c3-8883-467d-84cf-78a61c418733\" (UID: \"7191c5c3-8883-467d-84cf-78a61c418733\") " Jul 30 16:15:33.228303 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.228240 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twpqq\" (UniqueName: \"kubernetes.io/projected/7191c5c3-8883-467d-84cf-78a61c418733-kube-api-access-twpqq\") pod \"7191c5c3-8883-467d-84cf-78a61c418733\" (UID: \"7191c5c3-8883-467d-84cf-78a61c418733\") " Jul 30 16:15:33.228396 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.228304 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxfg5\" (UniqueName: \"kubernetes.io/projected/b920e676-ebf3-43de-b103-f24702a65710-kube-api-access-qxfg5\") pod \"b920e676-ebf3-43de-b103-f24702a65710\" (UID: \"b920e676-ebf3-43de-b103-f24702a65710\") " Jul 30 16:15:33.228508 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.228485 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b920e676-ebf3-43de-b103-f24702a65710-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b920e676-ebf3-43de-b103-f24702a65710" (UID: "b920e676-ebf3-43de-b103-f24702a65710"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 16:15:33.228580 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.228565 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b920e676-ebf3-43de-b103-f24702a65710-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:15:33.228632 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.228575 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7191c5c3-8883-467d-84cf-78a61c418733-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7191c5c3-8883-467d-84cf-78a61c418733" (UID: "7191c5c3-8883-467d-84cf-78a61c418733"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 16:15:33.230344 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.230321 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b920e676-ebf3-43de-b103-f24702a65710-kube-api-access-qxfg5" (OuterVolumeSpecName: "kube-api-access-qxfg5") pod "b920e676-ebf3-43de-b103-f24702a65710" (UID: "b920e676-ebf3-43de-b103-f24702a65710"). InnerVolumeSpecName "kube-api-access-qxfg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:15:33.230427 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.230323 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7191c5c3-8883-467d-84cf-78a61c418733-kube-api-access-twpqq" (OuterVolumeSpecName: "kube-api-access-twpqq") pod "7191c5c3-8883-467d-84cf-78a61c418733" (UID: "7191c5c3-8883-467d-84cf-78a61c418733"). InnerVolumeSpecName "kube-api-access-twpqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:15:33.276889 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.276791 2579 generic.go:356] "Generic (PLEG): container finished" podID="7191c5c3-8883-467d-84cf-78a61c418733" containerID="883948b109a097e331e9bf90e572b682abb93af889aa18485e634a5b4a440718" exitCode=0 Jul 30 16:15:33.276889 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.276860 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl" Jul 30 16:15:33.277042 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.276900 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl" event={"ID":"7191c5c3-8883-467d-84cf-78a61c418733","Type":"ContainerDied","Data":"883948b109a097e331e9bf90e572b682abb93af889aa18485e634a5b4a440718"} Jul 30 16:15:33.277042 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.276956 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl" event={"ID":"7191c5c3-8883-467d-84cf-78a61c418733","Type":"ContainerDied","Data":"b3272070bb7057fc258a972d598f7560127ad6660276002313ffe92d33d02502"} Jul 30 16:15:33.277042 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.276990 2579 scope.go:117] "RemoveContainer" containerID="883948b109a097e331e9bf90e572b682abb93af889aa18485e634a5b4a440718" Jul 30 16:15:33.278317 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.278294 2579 generic.go:356] "Generic (PLEG): container finished" podID="b920e676-ebf3-43de-b103-f24702a65710" containerID="d3bf73e29274b859a713e885d2d50a2c51924c26ca01c1e2fca42953d206f302" exitCode=0 Jul 30 16:15:33.278425 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.278319 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9" event={"ID":"b920e676-ebf3-43de-b103-f24702a65710","Type":"ContainerDied","Data":"d3bf73e29274b859a713e885d2d50a2c51924c26ca01c1e2fca42953d206f302"} Jul 30 16:15:33.278425 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.278333 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9" Jul 30 16:15:33.278425 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.278362 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9" event={"ID":"b920e676-ebf3-43de-b103-f24702a65710","Type":"ContainerDied","Data":"e3f35ef1a56af0c30b986f707a0b6294fc2b66c09340a1eaf22a0a3dce1313f5"} Jul 30 16:15:33.286976 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.286916 2579 scope.go:117] "RemoveContainer" containerID="883948b109a097e331e9bf90e572b682abb93af889aa18485e634a5b4a440718" Jul 30 16:15:33.287262 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:15:33.287234 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"883948b109a097e331e9bf90e572b682abb93af889aa18485e634a5b4a440718\": container with ID starting with 883948b109a097e331e9bf90e572b682abb93af889aa18485e634a5b4a440718 not found: ID does not exist" containerID="883948b109a097e331e9bf90e572b682abb93af889aa18485e634a5b4a440718" Jul 30 16:15:33.287342 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.287260 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"883948b109a097e331e9bf90e572b682abb93af889aa18485e634a5b4a440718"} err="failed to get container status \"883948b109a097e331e9bf90e572b682abb93af889aa18485e634a5b4a440718\": rpc error: code = NotFound desc = could not find container \"883948b109a097e331e9bf90e572b682abb93af889aa18485e634a5b4a440718\": container with ID starting with 883948b109a097e331e9bf90e572b682abb93af889aa18485e634a5b4a440718 not found: ID does not exist" Jul 30 16:15:33.287342 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.287278 2579 scope.go:117] "RemoveContainer" containerID="d3bf73e29274b859a713e885d2d50a2c51924c26ca01c1e2fca42953d206f302" Jul 30 16:15:33.301019 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.301000 2579 scope.go:117] "RemoveContainer" containerID="d3bf73e29274b859a713e885d2d50a2c51924c26ca01c1e2fca42953d206f302" Jul 30 16:15:33.301450 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:15:33.301429 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3bf73e29274b859a713e885d2d50a2c51924c26ca01c1e2fca42953d206f302\": container with ID starting with d3bf73e29274b859a713e885d2d50a2c51924c26ca01c1e2fca42953d206f302 not found: ID does not exist" containerID="d3bf73e29274b859a713e885d2d50a2c51924c26ca01c1e2fca42953d206f302" Jul 30 16:15:33.301525 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.301456 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3bf73e29274b859a713e885d2d50a2c51924c26ca01c1e2fca42953d206f302"} err="failed to get container status \"d3bf73e29274b859a713e885d2d50a2c51924c26ca01c1e2fca42953d206f302\": rpc error: code = NotFound desc = could not find container \"d3bf73e29274b859a713e885d2d50a2c51924c26ca01c1e2fca42953d206f302\": container with ID starting with d3bf73e29274b859a713e885d2d50a2c51924c26ca01c1e2fca42953d206f302 not found: ID does not exist" Jul 30 16:15:33.325141 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.325117 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9"] Jul 30 16:15:33.329915 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.329899 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7191c5c3-8883-467d-84cf-78a61c418733-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:15:33.329987 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.329918 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twpqq\" (UniqueName: \"kubernetes.io/projected/7191c5c3-8883-467d-84cf-78a61c418733-kube-api-access-twpqq\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:15:33.329987 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.329929 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxfg5\" (UniqueName: \"kubernetes.io/projected/b920e676-ebf3-43de-b103-f24702a65710-kube-api-access-qxfg5\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:15:33.338748 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.338729 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ae9970-859574cbcb-zrjs9"] Jul 30 16:15:33.366454 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.366433 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl"] Jul 30 16:15:33.390485 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:33.390463 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae9970-7cd7cbbdb6-wd5hl"] Jul 30 16:15:35.198093 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:35.198059 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7191c5c3-8883-467d-84cf-78a61c418733" path="/var/lib/kubelet/pods/7191c5c3-8883-467d-84cf-78a61c418733/volumes" Jul 30 16:15:35.198459 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:35.198418 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b920e676-ebf3-43de-b103-f24702a65710" path="/var/lib/kubelet/pods/b920e676-ebf3-43de-b103-f24702a65710/volumes" Jul 30 16:15:35.968327 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:35.968285 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:15:51.564145 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:51.564109 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2"] Jul 30 16:15:51.564575 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:51.564294 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2" podUID="03f2e58b-1567-44e4-8894-3d3ffa33304d" containerName="payload-processing" containerID="cri-o://4783de9b92b21cc3570df21a9f37f7943568afb610cd4b43630d3093ca6f3c11" gracePeriod=30 Jul 30 16:15:51.572170 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:51.571830 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm"] Jul 30 16:15:51.572170 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:51.572104 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm" podUID="735a5439-4f58-4176-ac4f-ec42abf7ba46" containerName="payload-pre-processing" containerID="cri-o://2296102e577187cbb7f6995192e1fb36e5ac7395ca38e1e97eb8e89deffa1217" gracePeriod=30 Jul 30 16:15:51.856549 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:51.856528 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2" Jul 30 16:15:51.859482 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:51.859462 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm" Jul 30 16:15:51.894290 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:51.894258 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735a5439-4f58-4176-ac4f-ec42abf7ba46-plugins-config-volume\") pod \"735a5439-4f58-4176-ac4f-ec42abf7ba46\" (UID: \"735a5439-4f58-4176-ac4f-ec42abf7ba46\") " Jul 30 16:15:51.894430 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:51.894325 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzblf\" (UniqueName: \"kubernetes.io/projected/735a5439-4f58-4176-ac4f-ec42abf7ba46-kube-api-access-pzblf\") pod \"735a5439-4f58-4176-ac4f-ec42abf7ba46\" (UID: \"735a5439-4f58-4176-ac4f-ec42abf7ba46\") " Jul 30 16:15:51.894430 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:51.894377 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03f2e58b-1567-44e4-8894-3d3ffa33304d-plugins-config-volume\") pod \"03f2e58b-1567-44e4-8894-3d3ffa33304d\" (UID: \"03f2e58b-1567-44e4-8894-3d3ffa33304d\") " Jul 30 16:15:51.894555 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:51.894478 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgjqq\" (UniqueName: \"kubernetes.io/projected/03f2e58b-1567-44e4-8894-3d3ffa33304d-kube-api-access-bgjqq\") pod \"03f2e58b-1567-44e4-8894-3d3ffa33304d\" (UID: \"03f2e58b-1567-44e4-8894-3d3ffa33304d\") " Jul 30 16:15:51.894906 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:51.894722 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/735a5439-4f58-4176-ac4f-ec42abf7ba46-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "735a5439-4f58-4176-ac4f-ec42abf7ba46" (UID: "735a5439-4f58-4176-ac4f-ec42abf7ba46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 16:15:51.895151 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:51.895126 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f2e58b-1567-44e4-8894-3d3ffa33304d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "03f2e58b-1567-44e4-8894-3d3ffa33304d" (UID: "03f2e58b-1567-44e4-8894-3d3ffa33304d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 16:15:51.897068 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:51.897038 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735a5439-4f58-4176-ac4f-ec42abf7ba46-kube-api-access-pzblf" (OuterVolumeSpecName: "kube-api-access-pzblf") pod "735a5439-4f58-4176-ac4f-ec42abf7ba46" (UID: "735a5439-4f58-4176-ac4f-ec42abf7ba46"). InnerVolumeSpecName "kube-api-access-pzblf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:15:51.897163 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:51.897095 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f2e58b-1567-44e4-8894-3d3ffa33304d-kube-api-access-bgjqq" (OuterVolumeSpecName: "kube-api-access-bgjqq") pod "03f2e58b-1567-44e4-8894-3d3ffa33304d" (UID: "03f2e58b-1567-44e4-8894-3d3ffa33304d"). InnerVolumeSpecName "kube-api-access-bgjqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:15:51.996218 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:51.996176 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgjqq\" (UniqueName: \"kubernetes.io/projected/03f2e58b-1567-44e4-8894-3d3ffa33304d-kube-api-access-bgjqq\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:15:51.996218 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:51.996211 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735a5439-4f58-4176-ac4f-ec42abf7ba46-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:15:51.996218 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:51.996220 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzblf\" (UniqueName: \"kubernetes.io/projected/735a5439-4f58-4176-ac4f-ec42abf7ba46-kube-api-access-pzblf\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:15:51.996218 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:51.996230 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03f2e58b-1567-44e4-8894-3d3ffa33304d-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:15:52.381001 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:52.380962 2579 generic.go:356] "Generic (PLEG): container finished" podID="03f2e58b-1567-44e4-8894-3d3ffa33304d" containerID="4783de9b92b21cc3570df21a9f37f7943568afb610cd4b43630d3093ca6f3c11" exitCode=0 Jul 30 16:15:52.381170 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:52.381020 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2" Jul 30 16:15:52.381170 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:52.381049 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2" event={"ID":"03f2e58b-1567-44e4-8894-3d3ffa33304d","Type":"ContainerDied","Data":"4783de9b92b21cc3570df21a9f37f7943568afb610cd4b43630d3093ca6f3c11"} Jul 30 16:15:52.381170 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:52.381140 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2" event={"ID":"03f2e58b-1567-44e4-8894-3d3ffa33304d","Type":"ContainerDied","Data":"591bc80f6feacd86016ce971eeef38d55d7e53cf1e75ce773f922f8546cb9762"} Jul 30 16:15:52.381170 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:52.381169 2579 scope.go:117] "RemoveContainer" containerID="4783de9b92b21cc3570df21a9f37f7943568afb610cd4b43630d3093ca6f3c11" Jul 30 16:15:52.382452 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:52.382429 2579 generic.go:356] "Generic (PLEG): container finished" podID="735a5439-4f58-4176-ac4f-ec42abf7ba46" containerID="2296102e577187cbb7f6995192e1fb36e5ac7395ca38e1e97eb8e89deffa1217" exitCode=0 Jul 30 16:15:52.382536 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:52.382459 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm" event={"ID":"735a5439-4f58-4176-ac4f-ec42abf7ba46","Type":"ContainerDied","Data":"2296102e577187cbb7f6995192e1fb36e5ac7395ca38e1e97eb8e89deffa1217"} Jul 30 16:15:52.382536 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:52.382494 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm" Jul 30 16:15:52.382536 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:52.382505 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm" event={"ID":"735a5439-4f58-4176-ac4f-ec42abf7ba46","Type":"ContainerDied","Data":"36f2345ab5200a302b3e6718b6454669113d362c9a7925e457e27cf920d23c70"} Jul 30 16:15:52.391076 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:52.391060 2579 scope.go:117] "RemoveContainer" containerID="4783de9b92b21cc3570df21a9f37f7943568afb610cd4b43630d3093ca6f3c11" Jul 30 16:15:52.391322 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:15:52.391303 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4783de9b92b21cc3570df21a9f37f7943568afb610cd4b43630d3093ca6f3c11\": container with ID starting with 4783de9b92b21cc3570df21a9f37f7943568afb610cd4b43630d3093ca6f3c11 not found: ID does not exist" containerID="4783de9b92b21cc3570df21a9f37f7943568afb610cd4b43630d3093ca6f3c11" Jul 30 16:15:52.391414 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:52.391332 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4783de9b92b21cc3570df21a9f37f7943568afb610cd4b43630d3093ca6f3c11"} err="failed to get container status \"4783de9b92b21cc3570df21a9f37f7943568afb610cd4b43630d3093ca6f3c11\": rpc error: code = NotFound desc = could not find container \"4783de9b92b21cc3570df21a9f37f7943568afb610cd4b43630d3093ca6f3c11\": container with ID starting with 4783de9b92b21cc3570df21a9f37f7943568afb610cd4b43630d3093ca6f3c11 not found: ID does not exist" Jul 30 16:15:52.391414 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:52.391355 2579 scope.go:117] "RemoveContainer" containerID="2296102e577187cbb7f6995192e1fb36e5ac7395ca38e1e97eb8e89deffa1217" Jul 30 16:15:52.399602 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:52.399586 2579 scope.go:117] "RemoveContainer" containerID="2296102e577187cbb7f6995192e1fb36e5ac7395ca38e1e97eb8e89deffa1217" Jul 30 16:15:52.399856 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:15:52.399837 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2296102e577187cbb7f6995192e1fb36e5ac7395ca38e1e97eb8e89deffa1217\": container with ID starting with 2296102e577187cbb7f6995192e1fb36e5ac7395ca38e1e97eb8e89deffa1217 not found: ID does not exist" containerID="2296102e577187cbb7f6995192e1fb36e5ac7395ca38e1e97eb8e89deffa1217" Jul 30 16:15:52.399919 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:52.399862 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2296102e577187cbb7f6995192e1fb36e5ac7395ca38e1e97eb8e89deffa1217"} err="failed to get container status \"2296102e577187cbb7f6995192e1fb36e5ac7395ca38e1e97eb8e89deffa1217\": rpc error: code = NotFound desc = could not find container \"2296102e577187cbb7f6995192e1fb36e5ac7395ca38e1e97eb8e89deffa1217\": container with ID starting with 2296102e577187cbb7f6995192e1fb36e5ac7395ca38e1e97eb8e89deffa1217 not found: ID does not exist" Jul 30 16:15:52.405740 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:52.405719 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2"] Jul 30 16:15:52.410082 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:52.410062 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f2cf4d-5d86b4778b-wt9h2"] Jul 30 16:15:52.420929 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:52.420908 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm"] Jul 30 16:15:52.426771 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:52.426753 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f2cf4d-64c5c4b8d8-hdgwm"] Jul 30 16:15:53.197267 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:53.197234 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03f2e58b-1567-44e4-8894-3d3ffa33304d" path="/var/lib/kubelet/pods/03f2e58b-1567-44e4-8894-3d3ffa33304d/volumes" Jul 30 16:15:53.197662 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:53.197647 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="735a5439-4f58-4176-ac4f-ec42abf7ba46" path="/var/lib/kubelet/pods/735a5439-4f58-4176-ac4f-ec42abf7ba46/volumes" Jul 30 16:15:54.839506 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:54.839471 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:15:58.885784 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:58.885743 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs"] Jul 30 16:15:58.886382 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:15:58.886002 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" podUID="afb02030-4a7c-4a7f-aa1f-a5308fe128c4" containerName="istio-proxy" containerID="cri-o://012ff6fa74e1a304f059498f194206143cbc497749e7f7850d4bb28f44ea0130" gracePeriod=30 Jul 30 16:16:04.143892 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.143858 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:16:04.215380 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.215301 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-workload-certs\") pod \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " Jul 30 16:16:04.215380 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.215355 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-envoy\") pod \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " Jul 30 16:16:04.215604 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.215394 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-token\") pod \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " Jul 30 16:16:04.215604 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.215418 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmlcz\" (UniqueName: \"kubernetes.io/projected/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-kube-api-access-pmlcz\") pod \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " Jul 30 16:16:04.215604 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.215447 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-credential-socket\") pod \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " Jul 30 16:16:04.215604 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.215469 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-workload-socket\") pod \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " Jul 30 16:16:04.215604 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.215495 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-data\") pod \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " Jul 30 16:16:04.215604 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.215535 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-podinfo\") pod \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " Jul 30 16:16:04.215604 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.215577 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istiod-ca-cert\") pod \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\" (UID: \"afb02030-4a7c-4a7f-aa1f-a5308fe128c4\") " Jul 30 16:16:04.216000 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.215598 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "afb02030-4a7c-4a7f-aa1f-a5308fe128c4" (UID: "afb02030-4a7c-4a7f-aa1f-a5308fe128c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:16:04.216000 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.215855 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "afb02030-4a7c-4a7f-aa1f-a5308fe128c4" (UID: "afb02030-4a7c-4a7f-aa1f-a5308fe128c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:16:04.216000 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.215871 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "afb02030-4a7c-4a7f-aa1f-a5308fe128c4" (UID: "afb02030-4a7c-4a7f-aa1f-a5308fe128c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:16:04.216155 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.216009 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-workload-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:04.216155 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.216032 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-workload-certs\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:04.216155 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.216045 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-credential-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:04.216155 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.216097 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "afb02030-4a7c-4a7f-aa1f-a5308fe128c4" (UID: "afb02030-4a7c-4a7f-aa1f-a5308fe128c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 16:16:04.216627 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.216603 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-data" (OuterVolumeSpecName: "istio-data") pod "afb02030-4a7c-4a7f-aa1f-a5308fe128c4" (UID: "afb02030-4a7c-4a7f-aa1f-a5308fe128c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:16:04.217674 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.217650 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "afb02030-4a7c-4a7f-aa1f-a5308fe128c4" (UID: "afb02030-4a7c-4a7f-aa1f-a5308fe128c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:16:04.217990 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.217971 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-kube-api-access-pmlcz" (OuterVolumeSpecName: "kube-api-access-pmlcz") pod "afb02030-4a7c-4a7f-aa1f-a5308fe128c4" (UID: "afb02030-4a7c-4a7f-aa1f-a5308fe128c4"). InnerVolumeSpecName "kube-api-access-pmlcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:16:04.218108 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.218089 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "afb02030-4a7c-4a7f-aa1f-a5308fe128c4" (UID: "afb02030-4a7c-4a7f-aa1f-a5308fe128c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 16:16:04.218171 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.218101 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "afb02030-4a7c-4a7f-aa1f-a5308fe128c4" (UID: "afb02030-4a7c-4a7f-aa1f-a5308fe128c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:16:04.317523 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.317487 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-envoy\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:04.317523 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.317515 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-token\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:04.317523 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.317529 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmlcz\" (UniqueName: \"kubernetes.io/projected/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-kube-api-access-pmlcz\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:04.317765 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.317540 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-data\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:04.317765 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.317551 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istio-podinfo\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:04.317765 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.317561 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb02030-4a7c-4a7f-aa1f-a5308fe128c4-istiod-ca-cert\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:04.446078 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.446040 2579 generic.go:356] "Generic (PLEG): container finished" podID="afb02030-4a7c-4a7f-aa1f-a5308fe128c4" containerID="012ff6fa74e1a304f059498f194206143cbc497749e7f7850d4bb28f44ea0130" exitCode=0 Jul 30 16:16:04.446253 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.446085 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" Jul 30 16:16:04.446253 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.446120 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" event={"ID":"afb02030-4a7c-4a7f-aa1f-a5308fe128c4","Type":"ContainerDied","Data":"012ff6fa74e1a304f059498f194206143cbc497749e7f7850d4bb28f44ea0130"} Jul 30 16:16:04.446253 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.446173 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs" event={"ID":"afb02030-4a7c-4a7f-aa1f-a5308fe128c4","Type":"ContainerDied","Data":"2892c93dd07ee8c69374fde9e13ad490dce6f3f34ba4c85dd654669c1755ac27"} Jul 30 16:16:04.446253 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.446197 2579 scope.go:117] "RemoveContainer" containerID="012ff6fa74e1a304f059498f194206143cbc497749e7f7850d4bb28f44ea0130" Jul 30 16:16:04.460115 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.456840 2579 scope.go:117] "RemoveContainer" containerID="1211e8cf0590ed48f5664d96b6e16282b6b4ec924f1abf9a451a27f6ab317bdb" Jul 30 16:16:04.466005 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.465982 2579 scope.go:117] "RemoveContainer" containerID="012ff6fa74e1a304f059498f194206143cbc497749e7f7850d4bb28f44ea0130" Jul 30 16:16:04.466238 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:16:04.466217 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"012ff6fa74e1a304f059498f194206143cbc497749e7f7850d4bb28f44ea0130\": container with ID starting with 012ff6fa74e1a304f059498f194206143cbc497749e7f7850d4bb28f44ea0130 not found: ID does not exist" containerID="012ff6fa74e1a304f059498f194206143cbc497749e7f7850d4bb28f44ea0130" Jul 30 16:16:04.466278 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.466244 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"012ff6fa74e1a304f059498f194206143cbc497749e7f7850d4bb28f44ea0130"} err="failed to get container status \"012ff6fa74e1a304f059498f194206143cbc497749e7f7850d4bb28f44ea0130\": rpc error: code = NotFound desc = could not find container \"012ff6fa74e1a304f059498f194206143cbc497749e7f7850d4bb28f44ea0130\": container with ID starting with 012ff6fa74e1a304f059498f194206143cbc497749e7f7850d4bb28f44ea0130 not found: ID does not exist" Jul 30 16:16:04.466278 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.466265 2579 scope.go:117] "RemoveContainer" containerID="1211e8cf0590ed48f5664d96b6e16282b6b4ec924f1abf9a451a27f6ab317bdb" Jul 30 16:16:04.466467 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:16:04.466450 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1211e8cf0590ed48f5664d96b6e16282b6b4ec924f1abf9a451a27f6ab317bdb\": container with ID starting with 1211e8cf0590ed48f5664d96b6e16282b6b4ec924f1abf9a451a27f6ab317bdb not found: ID does not exist" containerID="1211e8cf0590ed48f5664d96b6e16282b6b4ec924f1abf9a451a27f6ab317bdb" Jul 30 16:16:04.466511 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.466470 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1211e8cf0590ed48f5664d96b6e16282b6b4ec924f1abf9a451a27f6ab317bdb"} err="failed to get container status \"1211e8cf0590ed48f5664d96b6e16282b6b4ec924f1abf9a451a27f6ab317bdb\": rpc error: code = NotFound desc = could not find container \"1211e8cf0590ed48f5664d96b6e16282b6b4ec924f1abf9a451a27f6ab317bdb\": container with ID starting with 1211e8cf0590ed48f5664d96b6e16282b6b4ec924f1abf9a451a27f6ab317bdb not found: ID does not exist" Jul 30 16:16:04.472025 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.472001 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs"] Jul 30 16:16:04.475766 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:04.475743 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-f2cf4d-openshift-default-599c8f744-thjxs"] Jul 30 16:16:05.198014 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:05.197981 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afb02030-4a7c-4a7f-aa1f-a5308fe128c4" path="/var/lib/kubelet/pods/afb02030-4a7c-4a7f-aa1f-a5308fe128c4/volumes" Jul 30 16:16:09.898381 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:09.898347 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62"] Jul 30 16:16:09.898870 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:09.898608 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62" podUID="ade26a0e-afc0-45c9-8ded-8d84da122f2e" containerName="payload-processing" containerID="cri-o://d7b57372d85d25d0aef8544541c42684be5f8050447550464a80df9f32423120" gracePeriod=30 Jul 30 16:16:10.166965 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:10.166941 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62" Jul 30 16:16:10.271301 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:10.271264 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade26a0e-afc0-45c9-8ded-8d84da122f2e-plugins-config-volume\") pod \"ade26a0e-afc0-45c9-8ded-8d84da122f2e\" (UID: \"ade26a0e-afc0-45c9-8ded-8d84da122f2e\") " Jul 30 16:16:10.271463 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:10.271308 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqsjh\" (UniqueName: \"kubernetes.io/projected/ade26a0e-afc0-45c9-8ded-8d84da122f2e-kube-api-access-sqsjh\") pod \"ade26a0e-afc0-45c9-8ded-8d84da122f2e\" (UID: \"ade26a0e-afc0-45c9-8ded-8d84da122f2e\") " Jul 30 16:16:10.271652 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:10.271628 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade26a0e-afc0-45c9-8ded-8d84da122f2e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ade26a0e-afc0-45c9-8ded-8d84da122f2e" (UID: "ade26a0e-afc0-45c9-8ded-8d84da122f2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 16:16:10.273664 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:10.273639 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade26a0e-afc0-45c9-8ded-8d84da122f2e-kube-api-access-sqsjh" (OuterVolumeSpecName: "kube-api-access-sqsjh") pod "ade26a0e-afc0-45c9-8ded-8d84da122f2e" (UID: "ade26a0e-afc0-45c9-8ded-8d84da122f2e"). InnerVolumeSpecName "kube-api-access-sqsjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:16:10.372233 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:10.372201 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade26a0e-afc0-45c9-8ded-8d84da122f2e-plugins-config-volume\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:10.372233 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:10.372234 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqsjh\" (UniqueName: \"kubernetes.io/projected/ade26a0e-afc0-45c9-8ded-8d84da122f2e-kube-api-access-sqsjh\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:10.477020 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:10.476945 2579 generic.go:356] "Generic (PLEG): container finished" podID="ade26a0e-afc0-45c9-8ded-8d84da122f2e" containerID="d7b57372d85d25d0aef8544541c42684be5f8050447550464a80df9f32423120" exitCode=0 Jul 30 16:16:10.477020 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:10.476997 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62" Jul 30 16:16:10.477159 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:10.477025 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62" event={"ID":"ade26a0e-afc0-45c9-8ded-8d84da122f2e","Type":"ContainerDied","Data":"d7b57372d85d25d0aef8544541c42684be5f8050447550464a80df9f32423120"} Jul 30 16:16:10.477159 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:10.477079 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62" event={"ID":"ade26a0e-afc0-45c9-8ded-8d84da122f2e","Type":"ContainerDied","Data":"171c9d8c13580b62dec6c58d16093207cbf4642ed3a64f5282e2e1ea01024b1e"} Jul 30 16:16:10.477159 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:10.477100 2579 scope.go:117] "RemoveContainer" containerID="d7b57372d85d25d0aef8544541c42684be5f8050447550464a80df9f32423120" Jul 30 16:16:10.485875 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:10.485857 2579 scope.go:117] "RemoveContainer" containerID="d7b57372d85d25d0aef8544541c42684be5f8050447550464a80df9f32423120" Jul 30 16:16:10.486122 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:16:10.486099 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b57372d85d25d0aef8544541c42684be5f8050447550464a80df9f32423120\": container with ID starting with d7b57372d85d25d0aef8544541c42684be5f8050447550464a80df9f32423120 not found: ID does not exist" containerID="d7b57372d85d25d0aef8544541c42684be5f8050447550464a80df9f32423120" Jul 30 16:16:10.486174 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:10.486131 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b57372d85d25d0aef8544541c42684be5f8050447550464a80df9f32423120"} err="failed to get container status \"d7b57372d85d25d0aef8544541c42684be5f8050447550464a80df9f32423120\": rpc error: code = NotFound desc = could not find container \"d7b57372d85d25d0aef8544541c42684be5f8050447550464a80df9f32423120\": container with ID starting with d7b57372d85d25d0aef8544541c42684be5f8050447550464a80df9f32423120 not found: ID does not exist" Jul 30 16:16:10.498628 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:10.498603 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62"] Jul 30 16:16:10.502587 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:10.502569 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-5468b9-7d47c9996b-mlf62"] Jul 30 16:16:11.198035 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:11.197995 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ade26a0e-afc0-45c9-8ded-8d84da122f2e" path="/var/lib/kubelet/pods/ade26a0e-afc0-45c9-8ded-8d84da122f2e/volumes" Jul 30 16:16:12.840291 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:12.840259 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tjvjw"] Jul 30 16:16:16.951322 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:16.951283 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td"] Jul 30 16:16:16.951764 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:16.951571 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" podUID="03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" containerName="istio-proxy" containerID="cri-o://a38cb72716fa315f6e33e90903861202ba3fea82885e2f1d79cd5f4fade6a4b0" gracePeriod=30 Jul 30 16:16:22.201163 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.201140 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:16:22.277412 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.277386 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-token\") pod \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " Jul 30 16:16:22.277540 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.277424 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-credential-socket\") pod \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " Jul 30 16:16:22.277540 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.277458 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-data\") pod \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " Jul 30 16:16:22.277540 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.277473 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brl8d\" (UniqueName: \"kubernetes.io/projected/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-kube-api-access-brl8d\") pod \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " Jul 30 16:16:22.277540 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.277495 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-workload-certs\") pod \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " Jul 30 16:16:22.277540 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.277537 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-podinfo\") pod \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " Jul 30 16:16:22.277756 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.277561 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-workload-socket\") pod \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " Jul 30 16:16:22.277756 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.277597 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-envoy\") pod \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " Jul 30 16:16:22.277756 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.277661 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istiod-ca-cert\") pod \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\" (UID: \"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d\") " Jul 30 16:16:22.277756 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.277742 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" (UID: "03e3dbcd-ca05-4c95-a4e4-11cabd3f201d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:16:22.278016 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.277977 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" (UID: "03e3dbcd-ca05-4c95-a4e4-11cabd3f201d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:16:22.278178 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.278157 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-workload-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:22.278275 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.278183 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-credential-socket\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:22.278275 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.278157 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" (UID: "03e3dbcd-ca05-4c95-a4e4-11cabd3f201d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 16:16:22.278444 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.278416 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" (UID: "03e3dbcd-ca05-4c95-a4e4-11cabd3f201d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:16:22.278524 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.278433 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-data" (OuterVolumeSpecName: "istio-data") pod "03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" (UID: "03e3dbcd-ca05-4c95-a4e4-11cabd3f201d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:16:22.279763 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.279740 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-token" (OuterVolumeSpecName: "istio-token") pod "03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" (UID: "03e3dbcd-ca05-4c95-a4e4-11cabd3f201d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:16:22.279894 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.279871 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-kube-api-access-brl8d" (OuterVolumeSpecName: "kube-api-access-brl8d") pod "03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" (UID: "03e3dbcd-ca05-4c95-a4e4-11cabd3f201d"). InnerVolumeSpecName "kube-api-access-brl8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 16:16:22.279961 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.279911 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" (UID: "03e3dbcd-ca05-4c95-a4e4-11cabd3f201d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 16:16:22.280359 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.280341 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" (UID: "03e3dbcd-ca05-4c95-a4e4-11cabd3f201d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 16:16:22.379184 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.379162 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istiod-ca-cert\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:22.379184 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.379183 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-token\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:22.379326 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.379191 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-data\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:22.379326 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.379199 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brl8d\" (UniqueName: \"kubernetes.io/projected/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-kube-api-access-brl8d\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:22.379326 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.379208 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-workload-certs\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:22.379326 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.379216 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-podinfo\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:22.379326 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.379224 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d-istio-envoy\") on node \"ip-10-0-142-244.ec2.internal\" DevicePath \"\"" Jul 30 16:16:22.538549 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.538521 2579 generic.go:356] "Generic (PLEG): container finished" podID="03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" containerID="a38cb72716fa315f6e33e90903861202ba3fea82885e2f1d79cd5f4fade6a4b0" exitCode=0 Jul 30 16:16:22.538665 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.538571 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" Jul 30 16:16:22.538665 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.538605 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" event={"ID":"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d","Type":"ContainerDied","Data":"a38cb72716fa315f6e33e90903861202ba3fea82885e2f1d79cd5f4fade6a4b0"} Jul 30 16:16:22.538665 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.538657 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td" event={"ID":"03e3dbcd-ca05-4c95-a4e4-11cabd3f201d","Type":"ContainerDied","Data":"f0435726cfe3b427a2184fb153c9c021666a132d330683b507186b70c22a7309"} Jul 30 16:16:22.538792 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.538679 2579 scope.go:117] "RemoveContainer" containerID="a38cb72716fa315f6e33e90903861202ba3fea82885e2f1d79cd5f4fade6a4b0" Jul 30 16:16:22.548352 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.548336 2579 scope.go:117] "RemoveContainer" containerID="dbf3a76b03976437ce034872fb4ab054e48129633d19480d936a25e3131dbbf7" Jul 30 16:16:22.556125 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.556110 2579 scope.go:117] "RemoveContainer" containerID="a38cb72716fa315f6e33e90903861202ba3fea82885e2f1d79cd5f4fade6a4b0" Jul 30 16:16:22.556371 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:16:22.556353 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a38cb72716fa315f6e33e90903861202ba3fea82885e2f1d79cd5f4fade6a4b0\": container with ID starting with a38cb72716fa315f6e33e90903861202ba3fea82885e2f1d79cd5f4fade6a4b0 not found: ID does not exist" containerID="a38cb72716fa315f6e33e90903861202ba3fea82885e2f1d79cd5f4fade6a4b0" Jul 30 16:16:22.556423 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.556380 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a38cb72716fa315f6e33e90903861202ba3fea82885e2f1d79cd5f4fade6a4b0"} err="failed to get container status \"a38cb72716fa315f6e33e90903861202ba3fea82885e2f1d79cd5f4fade6a4b0\": rpc error: code = NotFound desc = could not find container \"a38cb72716fa315f6e33e90903861202ba3fea82885e2f1d79cd5f4fade6a4b0\": container with ID starting with a38cb72716fa315f6e33e90903861202ba3fea82885e2f1d79cd5f4fade6a4b0 not found: ID does not exist" Jul 30 16:16:22.556423 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.556399 2579 scope.go:117] "RemoveContainer" containerID="dbf3a76b03976437ce034872fb4ab054e48129633d19480d936a25e3131dbbf7" Jul 30 16:16:22.556615 ip-10-0-142-244 kubenswrapper[2579]: E0730 16:16:22.556597 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf3a76b03976437ce034872fb4ab054e48129633d19480d936a25e3131dbbf7\": container with ID starting with dbf3a76b03976437ce034872fb4ab054e48129633d19480d936a25e3131dbbf7 not found: ID does not exist" containerID="dbf3a76b03976437ce034872fb4ab054e48129633d19480d936a25e3131dbbf7" Jul 30 16:16:22.556683 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.556620 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf3a76b03976437ce034872fb4ab054e48129633d19480d936a25e3131dbbf7"} err="failed to get container status \"dbf3a76b03976437ce034872fb4ab054e48129633d19480d936a25e3131dbbf7\": rpc error: code = NotFound desc = could not find container \"dbf3a76b03976437ce034872fb4ab054e48129633d19480d936a25e3131dbbf7\": container with ID starting with dbf3a76b03976437ce034872fb4ab054e48129633d19480d936a25e3131dbbf7 not found: ID does not exist" Jul 30 16:16:22.561772 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.561747 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td"] Jul 30 16:16:22.565023 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:22.565004 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-5468b9-openshift-default-786db7548d-zd6td"] Jul 30 16:16:23.197683 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:23.197642 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" path="/var/lib/kubelet/pods/03e3dbcd-ca05-4c95-a4e4-11cabd3f201d/volumes" Jul 30 16:16:23.695854 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:23.695827 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-d7jmv_344cb966-68b6-4cb1-8206-3d822aece19b/manager/0.log" Jul 30 16:16:24.633983 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:24.633956 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56444d5c7b-jxwk5_7fa50e13-adca-4d63-9932-44713c9e0577/copy-manifests/0.log" Jul 30 16:16:24.646878 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:24.646843 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56444d5c7b-jxwk5_7fa50e13-adca-4d63-9932-44713c9e0577/manager/0.log" Jul 30 16:16:27.256252 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:27.256223 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-s8vct_e3b1403e-2d47-48fe-9195-330a8aeaa09e/manager/0.log" Jul 30 16:16:27.779526 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:27.779499 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-tjvjw_ae3de39f-55f2-4484-ba76-0f6a11077a1c/limitador/0.log" Jul 30 16:16:27.913120 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:27.913090 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-bqchp_a6c52501-2f20-40d2-8541-76d6285e51bc/manager/0.log" Jul 30 16:16:28.350948 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:28.350918 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf_b3fe5e6e-f812-4339-9345-481b4ee110c7/istio-proxy/0.log" Jul 30 16:16:28.875849 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:28.875815 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gkwnh_2915a001-9525-405e-a5cb-397ad774e66a/istio-proxy/1.log" Jul 30 16:16:29.280939 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:29.280862 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c8b769f9f-tfnst_04216c4f-4a2e-4c3b-a254-b2c11ad44244/router/0.log" Jul 30 16:16:29.833121 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:29.833091 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-5xhxq_bbf6af2f-2130-4af2-8790-d17c6a2eb6c1/main/0.log" Jul 30 16:16:30.083382 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:30.083309 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd45s8_fa820390-de40-4d51-904a-90c866d9917b/main/0.log" Jul 30 16:16:36.537056 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537024 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cv7vc/must-gather-2cp52"] Jul 30 16:16:36.537456 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537391 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f2e58b-1567-44e4-8894-3d3ffa33304d" containerName="payload-processing" Jul 30 16:16:36.537456 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537403 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f2e58b-1567-44e4-8894-3d3ffa33304d" containerName="payload-processing" Jul 30 16:16:36.537456 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537413 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afb02030-4a7c-4a7f-aa1f-a5308fe128c4" containerName="istio-proxy" Jul 30 16:16:36.537456 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537419 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="afb02030-4a7c-4a7f-aa1f-a5308fe128c4" containerName="istio-proxy" Jul 30 16:16:36.537456 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537426 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b920e676-ebf3-43de-b103-f24702a65710" containerName="payload-pre-processing" Jul 30 16:16:36.537456 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537433 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b920e676-ebf3-43de-b103-f24702a65710" containerName="payload-pre-processing" Jul 30 16:16:36.537456 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537439 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f51b26-8629-445f-aed8-4b3c208f46cd" containerName="payload-processing" Jul 30 16:16:36.537456 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537444 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f51b26-8629-445f-aed8-4b3c208f46cd" containerName="payload-processing" Jul 30 16:16:36.537456 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537454 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" containerName="istio-proxy" Jul 30 16:16:36.537456 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537459 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" containerName="istio-proxy" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537464 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5e95c65-c797-45c2-83f9-ae3939823ecb" containerName="istio-proxy" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537469 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e95c65-c797-45c2-83f9-ae3939823ecb" containerName="istio-proxy" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537478 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afb02030-4a7c-4a7f-aa1f-a5308fe128c4" containerName="istio-proxy" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537483 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="afb02030-4a7c-4a7f-aa1f-a5308fe128c4" containerName="istio-proxy" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537490 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="735a5439-4f58-4176-ac4f-ec42abf7ba46" containerName="payload-pre-processing" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537495 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="735a5439-4f58-4176-ac4f-ec42abf7ba46" containerName="payload-pre-processing" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537503 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade26a0e-afc0-45c9-8ded-8d84da122f2e" containerName="payload-processing" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537509 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade26a0e-afc0-45c9-8ded-8d84da122f2e" containerName="payload-processing" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537520 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7191c5c3-8883-467d-84cf-78a61c418733" containerName="payload-processing" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537526 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7191c5c3-8883-467d-84cf-78a61c418733" containerName="payload-processing" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537533 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="739747ed-95bb-4f2d-8b7b-8c0fb5025c65" containerName="payload-pre-processing" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537537 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="739747ed-95bb-4f2d-8b7b-8c0fb5025c65" containerName="payload-pre-processing" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537596 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="afb02030-4a7c-4a7f-aa1f-a5308fe128c4" containerName="istio-proxy" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537603 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="afb02030-4a7c-4a7f-aa1f-a5308fe128c4" containerName="istio-proxy" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537608 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5e95c65-c797-45c2-83f9-ae3939823ecb" containerName="istio-proxy" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537615 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4f51b26-8629-445f-aed8-4b3c208f46cd" containerName="payload-processing" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537623 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade26a0e-afc0-45c9-8ded-8d84da122f2e" containerName="payload-processing" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537628 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" containerName="istio-proxy" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537633 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" containerName="istio-proxy" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537638 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="739747ed-95bb-4f2d-8b7b-8c0fb5025c65" containerName="payload-pre-processing" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537644 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="03f2e58b-1567-44e4-8894-3d3ffa33304d" containerName="payload-processing" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537649 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="735a5439-4f58-4176-ac4f-ec42abf7ba46" containerName="payload-pre-processing" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537656 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b920e676-ebf3-43de-b103-f24702a65710" containerName="payload-pre-processing" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537662 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7191c5c3-8883-467d-84cf-78a61c418733" containerName="payload-processing" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537710 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" containerName="istio-proxy" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537716 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="03e3dbcd-ca05-4c95-a4e4-11cabd3f201d" containerName="istio-proxy" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537722 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5e95c65-c797-45c2-83f9-ae3939823ecb" containerName="istio-proxy" Jul 30 16:16:36.537774 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537727 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e95c65-c797-45c2-83f9-ae3939823ecb" containerName="istio-proxy" Jul 30 16:16:36.538651 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.537787 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5e95c65-c797-45c2-83f9-ae3939823ecb" containerName="istio-proxy" Jul 30 16:16:36.542678 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.542657 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cv7vc/must-gather-2cp52" Jul 30 16:16:36.545830 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.545780 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cv7vc\"/\"kube-root-ca.crt\"" Jul 30 16:16:36.545830 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.545822 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cv7vc\"/\"default-dockercfg-jszsr\"" Jul 30 16:16:36.545984 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.545850 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cv7vc\"/\"openshift-service-ca.crt\"" Jul 30 16:16:36.551342 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.551317 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cv7vc/must-gather-2cp52"] Jul 30 16:16:36.594152 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.594126 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c3485975-2600-46d3-aa26-6766d5b1f376-must-gather-output\") pod \"must-gather-2cp52\" (UID: \"c3485975-2600-46d3-aa26-6766d5b1f376\") " pod="openshift-must-gather-cv7vc/must-gather-2cp52" Jul 30 16:16:36.594265 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.594168 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hvws\" (UniqueName: \"kubernetes.io/projected/c3485975-2600-46d3-aa26-6766d5b1f376-kube-api-access-7hvws\") pod \"must-gather-2cp52\" (UID: \"c3485975-2600-46d3-aa26-6766d5b1f376\") " pod="openshift-must-gather-cv7vc/must-gather-2cp52" Jul 30 16:16:36.695245 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.695219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c3485975-2600-46d3-aa26-6766d5b1f376-must-gather-output\") pod \"must-gather-2cp52\" (UID: \"c3485975-2600-46d3-aa26-6766d5b1f376\") " pod="openshift-must-gather-cv7vc/must-gather-2cp52" Jul 30 16:16:36.695354 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.695257 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hvws\" (UniqueName: \"kubernetes.io/projected/c3485975-2600-46d3-aa26-6766d5b1f376-kube-api-access-7hvws\") pod \"must-gather-2cp52\" (UID: \"c3485975-2600-46d3-aa26-6766d5b1f376\") " pod="openshift-must-gather-cv7vc/must-gather-2cp52" Jul 30 16:16:36.695603 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.695585 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c3485975-2600-46d3-aa26-6766d5b1f376-must-gather-output\") pod \"must-gather-2cp52\" (UID: \"c3485975-2600-46d3-aa26-6766d5b1f376\") " pod="openshift-must-gather-cv7vc/must-gather-2cp52" Jul 30 16:16:36.703831 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.703786 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hvws\" (UniqueName: \"kubernetes.io/projected/c3485975-2600-46d3-aa26-6766d5b1f376-kube-api-access-7hvws\") pod \"must-gather-2cp52\" (UID: \"c3485975-2600-46d3-aa26-6766d5b1f376\") " pod="openshift-must-gather-cv7vc/must-gather-2cp52" Jul 30 16:16:36.852096 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.852034 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cv7vc/must-gather-2cp52" Jul 30 16:16:36.982838 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.982811 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cv7vc/must-gather-2cp52"] Jul 30 16:16:36.984769 ip-10-0-142-244 kubenswrapper[2579]: W0730 16:16:36.984743 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3485975_2600_46d3_aa26_6766d5b1f376.slice/crio-a050387419e440fec937c1e7fb327af239b02a2f725b97237dbf10e0440697fb WatchSource:0}: Error finding container a050387419e440fec937c1e7fb327af239b02a2f725b97237dbf10e0440697fb: Status 404 returned error can't find the container with id a050387419e440fec937c1e7fb327af239b02a2f725b97237dbf10e0440697fb Jul 30 16:16:36.986738 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:36.986722 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 16:16:37.613041 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:37.613003 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cv7vc/must-gather-2cp52" event={"ID":"c3485975-2600-46d3-aa26-6766d5b1f376","Type":"ContainerStarted","Data":"a050387419e440fec937c1e7fb327af239b02a2f725b97237dbf10e0440697fb"} Jul 30 16:16:39.626432 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:39.626396 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cv7vc/must-gather-2cp52" event={"ID":"c3485975-2600-46d3-aa26-6766d5b1f376","Type":"ContainerStarted","Data":"d90c5b18c9a5f228a9122be92dac5cf72fcf4ca91e9e8ab5f8ec4e57dd8623ad"} Jul 30 16:16:39.626916 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:39.626444 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cv7vc/must-gather-2cp52" event={"ID":"c3485975-2600-46d3-aa26-6766d5b1f376","Type":"ContainerStarted","Data":"50faf713837e8976e4caa395c758a3e1a3d6d6059ee7ac4a2b14751eb1aa4cad"} Jul 30 16:16:39.644578 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:39.644526 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cv7vc/must-gather-2cp52" podStartSLOduration=1.676238262 podStartE2EDuration="3.644508917s" totalImagesPullingTime="1.968270655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 16:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 16:16:39.642302849 +0000 UTC m=+4026.877433697" watchObservedRunningTime="2026-07-30 16:16:39.644508917 +0000 UTC m=+4026.879639763" Jul 30 16:16:40.567240 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:40.567213 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qh4bp_113d506f-fc3f-4ea8-9eb2-98a534c8b5db/global-pull-secret-syncer/0.log" Jul 30 16:16:40.740102 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:40.740045 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xj2rx_cd846133-3b4c-459a-abd2-58609374a7e8/konnectivity-agent/0.log" Jul 30 16:16:40.821236 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:40.821145 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-244.ec2.internal_27a35d60ff6bc9476dbc920db8c64fe2/haproxy/0.log" Jul 30 16:16:46.069522 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:46.069469 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-s8vct_e3b1403e-2d47-48fe-9195-330a8aeaa09e/manager/0.log" Jul 30 16:16:46.247061 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:46.247034 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-tjvjw_ae3de39f-55f2-4484-ba76-0f6a11077a1c/limitador/0.log" Jul 30 16:16:46.360951 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:46.360871 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-bqchp_a6c52501-2f20-40d2-8541-76d6285e51bc/manager/0.log" Jul 30 16:16:48.013823 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.013767 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_599c3a66-9f9e-42fa-8c2e-09c59a20a3ed/alertmanager/0.log" Jul 30 16:16:48.035955 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.035930 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_599c3a66-9f9e-42fa-8c2e-09c59a20a3ed/config-reloader/0.log" Jul 30 16:16:48.062752 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.062721 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_599c3a66-9f9e-42fa-8c2e-09c59a20a3ed/kube-rbac-proxy-web/0.log" Jul 30 16:16:48.092238 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.092186 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_599c3a66-9f9e-42fa-8c2e-09c59a20a3ed/kube-rbac-proxy/0.log" Jul 30 16:16:48.117296 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.117224 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_599c3a66-9f9e-42fa-8c2e-09c59a20a3ed/kube-rbac-proxy-metric/0.log" Jul 30 16:16:48.140005 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.139975 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_599c3a66-9f9e-42fa-8c2e-09c59a20a3ed/prom-label-proxy/0.log" Jul 30 16:16:48.169575 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.169488 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_599c3a66-9f9e-42fa-8c2e-09c59a20a3ed/init-config-reloader/0.log" Jul 30 16:16:48.239020 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.238939 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-pdxgg_ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9/kube-state-metrics/0.log" Jul 30 16:16:48.260582 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.260553 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-pdxgg_ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9/kube-rbac-proxy-main/0.log" Jul 30 16:16:48.281213 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.281144 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-pdxgg_ab5fd0b1-70ff-46cb-bc48-1a6dc61a75b9/kube-rbac-proxy-self/0.log" Jul 30 16:16:48.306865 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.306836 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-76889c7fb4-wm6zs_80bbd3b1-ac7f-4992-8196-766b644df1a6/metrics-server/0.log" Jul 30 16:16:48.371869 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.371840 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5clt4_e653e12c-86c8-4cca-897f-a7677c143087/node-exporter/0.log" Jul 30 16:16:48.391973 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.391947 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5clt4_e653e12c-86c8-4cca-897f-a7677c143087/kube-rbac-proxy/0.log" Jul 30 16:16:48.412790 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.412758 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5clt4_e653e12c-86c8-4cca-897f-a7677c143087/init-textfile/0.log" Jul 30 16:16:48.695953 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.695918 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dded69a3-a4db-4f1c-8102-ccc1539a4118/prometheus/0.log" Jul 30 16:16:48.726135 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.726107 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dded69a3-a4db-4f1c-8102-ccc1539a4118/config-reloader/0.log" Jul 30 16:16:48.768290 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.768258 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dded69a3-a4db-4f1c-8102-ccc1539a4118/thanos-sidecar/0.log" Jul 30 16:16:48.813769 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.813742 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dded69a3-a4db-4f1c-8102-ccc1539a4118/kube-rbac-proxy-web/0.log" Jul 30 16:16:48.863013 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.862936 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dded69a3-a4db-4f1c-8102-ccc1539a4118/kube-rbac-proxy/0.log" Jul 30 16:16:48.905215 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.905183 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dded69a3-a4db-4f1c-8102-ccc1539a4118/kube-rbac-proxy-thanos/0.log" Jul 30 16:16:48.949916 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.949882 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dded69a3-a4db-4f1c-8102-ccc1539a4118/init-config-reloader/0.log" Jul 30 16:16:48.989546 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:48.989513 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-ldkrv_3ab8732b-7ec2-4dd1-bd57-4c2907e2394e/prometheus-operator/0.log" Jul 30 16:16:49.016421 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:49.016376 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-ldkrv_3ab8732b-7ec2-4dd1-bd57-4c2907e2394e/kube-rbac-proxy/0.log" Jul 30 16:16:49.055619 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:49.055592 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-9t29f_bb7ed78c-9c26-4a78-b39e-ec702a794f7e/prometheus-operator-admission-webhook/0.log" Jul 30 16:16:49.087816 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:49.087766 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8466886869-jwkvt_f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6/telemeter-client/0.log" Jul 30 16:16:49.118656 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:49.118626 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8466886869-jwkvt_f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6/reload/0.log" Jul 30 16:16:49.149449 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:49.149420 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8466886869-jwkvt_f5c6a7b9-75b5-46dd-a5ac-a7449b3affd6/kube-rbac-proxy/0.log" Jul 30 16:16:51.184342 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.184305 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g"] Jul 30 16:16:51.191099 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.191070 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:51.201166 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.201137 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g"] Jul 30 16:16:51.365410 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.365350 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e83070e7-0b08-4784-be9e-f5843ebe184d-proc\") pod \"perf-node-gather-daemonset-qcd9g\" (UID: \"e83070e7-0b08-4784-be9e-f5843ebe184d\") " pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:51.365410 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.365407 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e83070e7-0b08-4784-be9e-f5843ebe184d-sys\") pod \"perf-node-gather-daemonset-qcd9g\" (UID: \"e83070e7-0b08-4784-be9e-f5843ebe184d\") " pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:51.365660 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.365537 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e83070e7-0b08-4784-be9e-f5843ebe184d-podres\") pod \"perf-node-gather-daemonset-qcd9g\" (UID: \"e83070e7-0b08-4784-be9e-f5843ebe184d\") " pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:51.365660 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.365620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlnf5\" (UniqueName: \"kubernetes.io/projected/e83070e7-0b08-4784-be9e-f5843ebe184d-kube-api-access-vlnf5\") pod \"perf-node-gather-daemonset-qcd9g\" (UID: \"e83070e7-0b08-4784-be9e-f5843ebe184d\") " pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:51.365660 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.365649 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e83070e7-0b08-4784-be9e-f5843ebe184d-lib-modules\") pod \"perf-node-gather-daemonset-qcd9g\" (UID: \"e83070e7-0b08-4784-be9e-f5843ebe184d\") " pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:51.466641 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.466561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e83070e7-0b08-4784-be9e-f5843ebe184d-podres\") pod \"perf-node-gather-daemonset-qcd9g\" (UID: \"e83070e7-0b08-4784-be9e-f5843ebe184d\") " pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:51.466641 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.466629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlnf5\" (UniqueName: \"kubernetes.io/projected/e83070e7-0b08-4784-be9e-f5843ebe184d-kube-api-access-vlnf5\") pod \"perf-node-gather-daemonset-qcd9g\" (UID: \"e83070e7-0b08-4784-be9e-f5843ebe184d\") " pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:51.466892 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.466651 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e83070e7-0b08-4784-be9e-f5843ebe184d-lib-modules\") pod \"perf-node-gather-daemonset-qcd9g\" (UID: \"e83070e7-0b08-4784-be9e-f5843ebe184d\") " pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:51.466892 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.466692 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e83070e7-0b08-4784-be9e-f5843ebe184d-proc\") pod \"perf-node-gather-daemonset-qcd9g\" (UID: \"e83070e7-0b08-4784-be9e-f5843ebe184d\") " pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:51.466892 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.466708 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e83070e7-0b08-4784-be9e-f5843ebe184d-sys\") pod \"perf-node-gather-daemonset-qcd9g\" (UID: \"e83070e7-0b08-4784-be9e-f5843ebe184d\") " pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:51.466892 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.466743 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e83070e7-0b08-4784-be9e-f5843ebe184d-podres\") pod \"perf-node-gather-daemonset-qcd9g\" (UID: \"e83070e7-0b08-4784-be9e-f5843ebe184d\") " pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:51.466892 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.466812 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e83070e7-0b08-4784-be9e-f5843ebe184d-proc\") pod \"perf-node-gather-daemonset-qcd9g\" (UID: \"e83070e7-0b08-4784-be9e-f5843ebe184d\") " pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:51.466892 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.466841 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e83070e7-0b08-4784-be9e-f5843ebe184d-lib-modules\") pod \"perf-node-gather-daemonset-qcd9g\" (UID: \"e83070e7-0b08-4784-be9e-f5843ebe184d\") " pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:51.466892 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.466823 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e83070e7-0b08-4784-be9e-f5843ebe184d-sys\") pod \"perf-node-gather-daemonset-qcd9g\" (UID: \"e83070e7-0b08-4784-be9e-f5843ebe184d\") " pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:51.476197 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.476165 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlnf5\" (UniqueName: \"kubernetes.io/projected/e83070e7-0b08-4784-be9e-f5843ebe184d-kube-api-access-vlnf5\") pod \"perf-node-gather-daemonset-qcd9g\" (UID: \"e83070e7-0b08-4784-be9e-f5843ebe184d\") " pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:51.510490 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.510247 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:51.681224 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.680827 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g"] Jul 30 16:16:51.683653 ip-10-0-142-244 kubenswrapper[2579]: W0730 16:16:51.683624 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode83070e7_0b08_4784_be9e_f5843ebe184d.slice/crio-04f827bc62c3a46cdd418f806164f6eeb53a79db31d0565ee2ffa30228221429 WatchSource:0}: Error finding container 04f827bc62c3a46cdd418f806164f6eeb53a79db31d0565ee2ffa30228221429: Status 404 returned error can't find the container with id 04f827bc62c3a46cdd418f806164f6eeb53a79db31d0565ee2ffa30228221429 Jul 30 16:16:51.721827 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:51.721746 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" event={"ID":"e83070e7-0b08-4784-be9e-f5843ebe184d","Type":"ContainerStarted","Data":"04f827bc62c3a46cdd418f806164f6eeb53a79db31d0565ee2ffa30228221429"} Jul 30 16:16:52.729495 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:52.729457 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" event={"ID":"e83070e7-0b08-4784-be9e-f5843ebe184d","Type":"ContainerStarted","Data":"a6f6b34143932c5dfa19a94f285eea9623494dc5971af4b6c7f79d9e058c13e4"} Jul 30 16:16:52.732283 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:52.729570 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:52.751445 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:52.751382 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" podStartSLOduration=1.751361942 podStartE2EDuration="1.751361942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 16:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 16:16:52.749965965 +0000 UTC m=+4039.985096822" watchObservedRunningTime="2026-07-30 16:16:52.751361942 +0000 UTC m=+4039.986492788" Jul 30 16:16:53.565288 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:53.565259 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gq5ph_829b9cdd-657c-4f62-8a83-813c43eefa09/dns/0.log" Jul 30 16:16:53.585155 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:53.585124 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gq5ph_829b9cdd-657c-4f62-8a83-813c43eefa09/kube-rbac-proxy/0.log" Jul 30 16:16:53.702939 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:53.702914 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cvbfc_c35ba36d-8199-471f-99df-70cc5ba0c05e/dns-node-resolver/0.log" Jul 30 16:16:54.354255 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:54.354226 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g8lsb_ec9478e9-d9e0-4ea7-9412-e36fc1a305c7/node-ca/0.log" Jul 30 16:16:55.361485 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:55.361443 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf_b3fe5e6e-f812-4339-9345-481b4ee110c7/istio-proxy/0.log" Jul 30 16:16:55.595251 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:55.595033 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gkwnh_2915a001-9525-405e-a5cb-397ad774e66a/istio-proxy/0.log" Jul 30 16:16:55.630066 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:55.630038 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gkwnh_2915a001-9525-405e-a5cb-397ad774e66a/istio-proxy/1.log" Jul 30 16:16:55.709219 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:55.709159 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c8b769f9f-tfnst_04216c4f-4a2e-4c3b-a254-b2c11ad44244/router/0.log" Jul 30 16:16:56.526432 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:56.526399 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ksbjt_d5098562-bcb2-4825-8429-184d57da5659/serve-healthcheck-canary/0.log" Jul 30 16:16:57.088981 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:57.088948 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8tgkr_453cbb90-b8f9-48e1-981d-22a44831433d/kube-rbac-proxy/0.log" Jul 30 16:16:57.111837 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:57.111784 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8tgkr_453cbb90-b8f9-48e1-981d-22a44831433d/exporter/0.log" Jul 30 16:16:57.135040 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:57.135013 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8tgkr_453cbb90-b8f9-48e1-981d-22a44831433d/extractor/0.log" Jul 30 16:16:58.747325 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:58.747293 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cv7vc/perf-node-gather-daemonset-qcd9g" Jul 30 16:16:59.517053 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:16:59.516936 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-d7jmv_344cb966-68b6-4cb1-8206-3d822aece19b/manager/0.log" Jul 30 16:17:00.195891 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:00.195861 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56444d5c7b-jxwk5_7fa50e13-adca-4d63-9932-44713c9e0577/manager/0.log" Jul 30 16:17:00.214678 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:00.214651 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56444d5c7b-jxwk5_7fa50e13-adca-4d63-9932-44713c9e0577/copy-manifests/0.log" Jul 30 16:17:00.597098 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:00.597039 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf_b3fe5e6e-f812-4339-9345-481b4ee110c7/istio-proxy/0.log" Jul 30 16:17:00.791227 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:00.791141 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gkwnh_2915a001-9525-405e-a5cb-397ad774e66a/istio-proxy/0.log" Jul 30 16:17:00.835522 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:00.835489 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gkwnh_2915a001-9525-405e-a5cb-397ad774e66a/istio-proxy/1.log" Jul 30 16:17:00.895078 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:00.895048 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c8b769f9f-tfnst_04216c4f-4a2e-4c3b-a254-b2c11ad44244/router/0.log" Jul 30 16:17:01.588705 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:01.588619 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-556f467f75-szpt6_12fbfdca-4ab3-4f52-84aa-430978a4ee68/manager/0.log" Jul 30 16:17:02.309019 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:02.308987 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-hchlf_b3fe5e6e-f812-4339-9345-481b4ee110c7/istio-proxy/0.log" Jul 30 16:17:02.512399 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:02.512372 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gkwnh_2915a001-9525-405e-a5cb-397ad774e66a/istio-proxy/0.log" Jul 30 16:17:02.551059 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:02.551002 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gkwnh_2915a001-9525-405e-a5cb-397ad774e66a/istio-proxy/1.log" Jul 30 16:17:02.607067 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:02.607001 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c8b769f9f-tfnst_04216c4f-4a2e-4c3b-a254-b2c11ad44244/router/0.log" Jul 30 16:17:06.421205 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:06.421170 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-tqn4m_ed272483-e9f3-48ad-8cb0-a79b10f634dd/migrator/0.log" Jul 30 16:17:06.441737 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:06.441708 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-tqn4m_ed272483-e9f3-48ad-8cb0-a79b10f634dd/graceful-termination/0.log" Jul 30 16:17:06.801075 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:06.800994 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q_0cc61754-0414-4c86-84de-250ba68dc560/kube-storage-version-migrator-operator/1.log" Jul 30 16:17:06.801931 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:06.801911 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-2d49q_0cc61754-0414-4c86-84de-250ba68dc560/kube-storage-version-migrator-operator/0.log" Jul 30 16:17:08.223248 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:08.223172 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbrvz_e04c52a8-91f2-49a2-9e17-c015e0c44d36/kube-multus-additional-cni-plugins/0.log" Jul 30 16:17:08.242488 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:08.242468 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbrvz_e04c52a8-91f2-49a2-9e17-c015e0c44d36/egress-router-binary-copy/0.log" Jul 30 16:17:08.262339 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:08.262312 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbrvz_e04c52a8-91f2-49a2-9e17-c015e0c44d36/cni-plugins/0.log" Jul 30 16:17:08.282914 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:08.282885 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbrvz_e04c52a8-91f2-49a2-9e17-c015e0c44d36/bond-cni-plugin/0.log" Jul 30 16:17:08.303293 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:08.303261 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbrvz_e04c52a8-91f2-49a2-9e17-c015e0c44d36/routeoverride-cni/0.log" Jul 30 16:17:08.323955 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:08.323840 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbrvz_e04c52a8-91f2-49a2-9e17-c015e0c44d36/whereabouts-cni-bincopy/0.log" Jul 30 16:17:08.343971 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:08.343947 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbrvz_e04c52a8-91f2-49a2-9e17-c015e0c44d36/whereabouts-cni/0.log" Jul 30 16:17:08.424524 ip-10-0-142-244 kubenswrapper[2579]: I0730 16:17:08.424489 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-q574q_f3999367-09f0-4979-8867-d16af22db8f6/kube-multus/0.log"