Jul 31 14:12:27.860141 ip-10-0-132-38 systemd[1]: Starting Kubernetes Kubelet... Jul 31 14:12:28.199759 ip-10-0-132-38 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 14:12:28.199759 ip-10-0-132-38 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 31 14:12:28.199759 ip-10-0-132-38 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 14:12:28.199759 ip-10-0-132-38 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 31 14:12:28.199759 ip-10-0-132-38 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 14:12:28.201849 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.201752 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 31 14:12:28.208203 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208187 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 14:12:28.208203 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208203 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208207 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208211 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208214 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208217 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208220 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208223 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208226 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208229 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208233 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208236 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208239 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208242 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208244 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208247 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208250 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208252 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208255 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208257 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208260 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 14:12:28.208268 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208262 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208265 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208268 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208270 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208273 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208276 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208279 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208282 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208284 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208287 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208290 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208293 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208295 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208299 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208301 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208304 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208306 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208309 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208311 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 14:12:28.208752 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208314 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208316 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208319 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208322 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208327 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208330 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208333 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208336 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208338 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208341 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208343 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208346 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208348 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208351 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208354 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208357 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208359 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208362 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208365 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208367 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 14:12:28.209230 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208370 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208373 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208377 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208380 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208382 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208385 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208388 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208390 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208393 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208395 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208398 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208400 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208403 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208405 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208408 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208410 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208413 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208416 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208419 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208422 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 14:12:28.209754 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208424 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208427 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208429 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208432 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208436 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208438 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208441 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208443 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208446 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208450 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208454 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208457 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208460 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208463 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208466 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208468 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208471 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208474 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.208477 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209192 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 14:12:28.210247 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209199 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209202 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209205 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209208 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209211 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209215 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209218 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209221 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209223 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209227 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209230 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209232 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209235 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209238 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209241 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209244 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209247 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209250 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209253 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209256 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 14:12:28.210720 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209259 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209261 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209264 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209267 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209270 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209274 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209278 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209281 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209284 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209286 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209289 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209292 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209297 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209300 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209303 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209306 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209309 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209312 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209315 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 14:12:28.211243 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209318 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209321 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209323 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209326 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209329 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209332 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209334 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209336 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209339 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209342 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209345 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209347 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209350 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209353 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209355 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209358 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209360 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209363 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209365 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 14:12:28.211722 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209368 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209370 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209372 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209375 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209377 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209380 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209382 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209385 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209387 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209390 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209392 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209395 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209398 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209400 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209403 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209405 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209408 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209411 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209414 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209416 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 14:12:28.212316 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209420 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209423 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209426 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209428 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209431 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209435 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209438 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209441 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209443 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209446 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209448 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209451 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209454 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209456 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209459 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209461 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209464 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209467 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209470 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209472 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 14:12:28.212791 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209832 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209837 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209862 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209868 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209871 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209874 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209876 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209879 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209882 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209885 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209887 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209890 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209892 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209895 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209897 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209900 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209903 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209906 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209910 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209913 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 14:12:28.213292 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209916 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209918 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209921 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209923 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209926 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209928 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209931 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209933 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209937 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209941 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209944 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209947 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209949 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209952 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209955 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209958 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209961 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209964 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209966 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209969 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 14:12:28.213779 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209971 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209974 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209977 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209980 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209982 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209985 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209987 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209990 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209992 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209995 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.209997 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210000 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210002 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210005 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210008 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210010 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210013 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210016 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210019 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210022 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 14:12:28.214290 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210024 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210027 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210029 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210032 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210034 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210037 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210040 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210043 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210045 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210048 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210051 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210055 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210058 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210061 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210064 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210070 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210074 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210076 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210079 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 14:12:28.214810 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210082 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210084 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210087 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210090 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210092 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210095 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210097 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210100 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210103 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210105 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210108 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210110 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210113 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210115 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210118 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210120 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210123 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210125 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210129 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.210132 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 14:12:28.215291 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210205 2577 flags.go:64] FLAG: --address="0.0.0.0" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210211 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210665 2577 flags.go:64] FLAG: --anonymous-auth="true" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210670 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210675 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210678 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210682 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210686 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210689 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210693 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210697 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210700 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210703 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210706 2577 flags.go:64] FLAG: --cgroup-root="" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210709 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210712 2577 flags.go:64] FLAG: --client-ca-file="" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210715 2577 flags.go:64] FLAG: --cloud-provider="external" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210718 2577 flags.go:64] FLAG: --cluster-dns="[]" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210722 2577 flags.go:64] FLAG: --cluster-domain="" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210726 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210729 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210732 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210735 2577 flags.go:64] FLAG: --container-log-max-files="5" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210739 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 31 14:12:28.215777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210742 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210745 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210749 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210752 2577 flags.go:64] FLAG: --contention-profiling="false" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210755 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210757 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210763 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210766 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210770 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210774 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210777 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210780 2577 flags.go:64] FLAG: --enable-load-reader="false" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210783 2577 flags.go:64] FLAG: --enable-server="true" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210786 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210790 2577 flags.go:64] FLAG: --event-burst="100" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210793 2577 flags.go:64] FLAG: --event-qps="50" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210796 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210799 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210802 2577 flags.go:64] FLAG: --eviction-hard="" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210806 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210809 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210812 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210815 2577 flags.go:64] FLAG: --eviction-soft="" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210817 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210820 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 31 14:12:28.216362 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210823 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210826 2577 flags.go:64] FLAG: --experimental-mounter-path="" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210829 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210832 2577 flags.go:64] FLAG: --fail-swap-on="true" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210835 2577 flags.go:64] FLAG: --feature-gates="" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210851 2577 flags.go:64] FLAG: --file-check-frequency="20s" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210855 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210858 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210861 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210864 2577 flags.go:64] FLAG: --healthz-port="10248" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210867 2577 flags.go:64] FLAG: --help="false" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210871 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-132-38.ec2.internal" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210875 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210879 2577 flags.go:64] FLAG: --http-check-frequency="20s" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210882 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210886 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210889 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210892 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210895 2577 flags.go:64] FLAG: --image-service-endpoint="" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210898 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210901 2577 flags.go:64] FLAG: --kube-api-burst="100" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210904 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210908 2577 flags.go:64] FLAG: --kube-api-qps="50" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210910 2577 flags.go:64] FLAG: --kube-reserved="" Jul 31 14:12:28.216994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210913 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210916 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210920 2577 flags.go:64] FLAG: --kubelet-cgroups="" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210923 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210926 2577 flags.go:64] FLAG: --lock-file="" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210929 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210931 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210934 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210939 2577 flags.go:64] FLAG: --log-json-split-stream="false" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210942 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210945 2577 flags.go:64] FLAG: --log-text-split-stream="false" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210948 2577 flags.go:64] FLAG: --logging-format="text" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210951 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210954 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210957 2577 flags.go:64] FLAG: --manifest-url="" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210960 2577 flags.go:64] FLAG: --manifest-url-header="" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210964 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210967 2577 flags.go:64] FLAG: --max-open-files="1000000" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210971 2577 flags.go:64] FLAG: --max-pods="110" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210976 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210979 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210983 2577 flags.go:64] FLAG: --memory-manager-policy="None" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210986 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210989 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210992 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 31 14:12:28.217861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.210995 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211002 2577 flags.go:64] FLAG: --node-status-max-images="50" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211006 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211009 2577 flags.go:64] FLAG: --oom-score-adj="-999" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211012 2577 flags.go:64] FLAG: --pod-cidr="" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211015 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211021 2577 flags.go:64] FLAG: --pod-manifest-path="" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211024 2577 flags.go:64] FLAG: --pod-max-pids="-1" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211027 2577 flags.go:64] FLAG: --pods-per-core="0" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211030 2577 flags.go:64] FLAG: --port="10250" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211033 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211036 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0473e0b925d4989c9" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211039 2577 flags.go:64] FLAG: --qos-reserved="" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211042 2577 flags.go:64] FLAG: --read-only-port="10255" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211045 2577 flags.go:64] FLAG: --register-node="true" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211048 2577 flags.go:64] FLAG: --register-with-taints="" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211052 2577 flags.go:64] FLAG: --registry-burst="10" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211054 2577 flags.go:64] FLAG: --registry-qps="5" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211057 2577 flags.go:64] FLAG: --reserved-cpus="" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211060 2577 flags.go:64] FLAG: --reserved-memory="" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211064 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211067 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211070 2577 flags.go:64] FLAG: --rotate-certificates="false" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211072 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211075 2577 flags.go:64] FLAG: --runonce="false" Jul 31 14:12:28.219008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211078 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211082 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211085 2577 flags.go:64] FLAG: --seccomp-default="false" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211088 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211092 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211095 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211098 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211101 2577 flags.go:64] FLAG: --storage-driver-password="root" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211104 2577 flags.go:64] FLAG: --storage-driver-secure="false" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211107 2577 flags.go:64] FLAG: --storage-driver-table="stats" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211110 2577 flags.go:64] FLAG: --storage-driver-user="root" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211113 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211116 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211119 2577 flags.go:64] FLAG: --system-cgroups="" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211121 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211127 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211129 2577 flags.go:64] FLAG: --tls-cert-file="" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211132 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211137 2577 flags.go:64] FLAG: --tls-min-version="" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211140 2577 flags.go:64] FLAG: --tls-private-key-file="" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211143 2577 flags.go:64] FLAG: --topology-manager-policy="none" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211146 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211148 2577 flags.go:64] FLAG: --topology-manager-scope="container" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211151 2577 flags.go:64] FLAG: --v="2" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211155 2577 flags.go:64] FLAG: --version="false" Jul 31 14:12:28.220032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211159 2577 flags.go:64] FLAG: --vmodule="" Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211163 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.211166 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211299 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211303 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211306 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211309 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211312 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211317 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211320 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211323 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211327 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211330 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211333 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211336 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211339 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211342 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211345 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211349 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211353 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 14:12:28.220824 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211356 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211359 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211362 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211364 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211367 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211369 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211372 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211375 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211378 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211380 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211383 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211386 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211388 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211391 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211393 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211396 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211399 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211401 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211404 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 14:12:28.221698 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211406 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211410 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211412 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211415 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211420 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211423 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211425 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211428 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211430 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211434 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211436 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211438 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211441 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211444 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211446 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211449 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211452 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211454 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211457 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211459 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 14:12:28.222319 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211467 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211470 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211473 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211475 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211477 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211480 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211484 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211488 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211490 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211493 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211496 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211499 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211502 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211506 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211509 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211512 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211516 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211519 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211522 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 14:12:28.222808 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211524 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211527 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211530 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211533 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211535 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211539 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211542 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211546 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211548 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211551 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211554 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211557 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211559 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211562 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211570 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211573 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211576 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211579 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211581 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 14:12:28.223296 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211584 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 14:12:28.223761 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211586 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 14:12:28.223761 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211589 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 14:12:28.223761 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211592 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 14:12:28.223761 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.211594 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 14:12:28.223761 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.212261 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 14:12:28.223761 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.219877 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 31 14:12:28.223761 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.219897 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 31 14:12:28.223761 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.220416 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 31 14:12:28.223761 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.220432 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 31 14:12:28.223761 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220537 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 14:12:28.223761 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220545 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 14:12:28.223761 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220550 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 14:12:28.223761 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220555 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 14:12:28.223761 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220560 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 14:12:28.223761 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220566 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220572 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220577 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220582 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220586 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220590 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220594 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220599 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220603 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220608 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220612 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220616 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220620 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220625 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220629 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220633 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220637 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220642 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220646 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220651 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 14:12:28.224211 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220655 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220660 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220664 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220668 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220726 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220903 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220912 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220917 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220923 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220928 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220933 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220938 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220943 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220948 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220952 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220956 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220961 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220966 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220977 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 14:12:28.224703 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220982 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220987 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220992 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.220996 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221001 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221005 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221010 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221015 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221019 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221024 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221028 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221038 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221042 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221047 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221052 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221057 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221061 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221067 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221074 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221079 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 14:12:28.225180 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221083 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221088 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221092 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221097 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221106 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221110 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221115 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221119 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221123 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221128 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221133 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221137 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221142 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221147 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221151 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221156 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221163 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221174 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221179 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 14:12:28.225676 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221183 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 14:12:28.226220 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221188 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 14:12:28.226220 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221193 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 14:12:28.226220 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221197 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 14:12:28.226220 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221201 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 14:12:28.226220 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221206 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 14:12:28.226220 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221211 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 14:12:28.226220 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221216 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 14:12:28.226220 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221220 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 14:12:28.226220 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221225 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 14:12:28.226220 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221235 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 14:12:28.226220 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221240 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 14:12:28.226220 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221245 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 14:12:28.226220 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221250 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 14:12:28.226220 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221255 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 14:12:28.226220 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221259 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 14:12:28.226590 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.221268 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 14:12:28.226590 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221776 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 14:12:28.226590 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221786 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 14:12:28.226590 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221791 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 14:12:28.226590 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221801 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 14:12:28.226590 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221806 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 14:12:28.226590 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221811 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 14:12:28.226590 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221816 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 14:12:28.226590 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221821 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 14:12:28.226590 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221825 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 14:12:28.226590 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221830 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 14:12:28.226590 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221834 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 14:12:28.226590 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221856 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 14:12:28.226590 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221864 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221869 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221873 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221884 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221892 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221898 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221903 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221908 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221912 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221917 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221921 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221926 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221930 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221935 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221941 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221950 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221963 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221969 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221973 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221978 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 14:12:28.226956 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221983 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.221987 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222033 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222162 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222166 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222170 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222173 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222175 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222178 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222181 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222184 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222187 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222190 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222193 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222196 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222199 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222202 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222205 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222207 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222210 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 14:12:28.227434 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222213 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222216 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222219 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222221 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222225 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222228 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222230 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222233 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222236 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222238 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222241 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222246 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222248 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222251 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222254 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222257 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222262 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222265 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222268 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222271 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 14:12:28.227939 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222274 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222277 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222280 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222283 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222285 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222288 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222291 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222294 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222296 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222299 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222302 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222304 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222307 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222310 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222313 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222316 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222319 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222322 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222327 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222330 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 14:12:28.228445 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222335 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 14:12:28.228966 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222337 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 14:12:28.228966 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222342 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 14:12:28.228966 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222344 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 14:12:28.228966 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222349 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 14:12:28.228966 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222352 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 14:12:28.228966 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:28.222355 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 14:12:28.228966 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.222365 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 14:12:28.228966 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.222495 2577 server.go:962] "Client rotation is on, will bootstrap in background" Jul 31 14:12:28.228966 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.224331 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 31 14:12:28.228966 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.225545 2577 server.go:1019] "Starting client certificate rotation" Jul 31 14:12:28.228966 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.225644 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 31 14:12:28.228966 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.225679 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 31 14:12:28.228966 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.227993 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 31 14:12:28.229346 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.229114 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 31 14:12:28.242094 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.242075 2577 log.go:25] "Validated CRI v1 runtime API" Jul 31 14:12:28.244060 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.244044 2577 log.go:25] "Validated CRI v1 image API" Jul 31 14:12:28.244833 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.244819 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 31 14:12:28.248290 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.248271 2577 fs.go:135] Filesystem UUIDs: map[091c6366-c92e-463d-a5a0-5f4396c63d5c:/dev/nvme0n1p4 6d94a261-48ac-4af7-9d8e-a65c80e3e941:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 31 14:12:28.248365 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.248289 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 31 14:12:28.253568 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.253463 2577 manager.go:216] Machine: {Timestamp:2026-07-31 14:12:28.252309641 +0000 UTC m=+0.289132702 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3052636 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2285ee83a796651629da4634b12006 SystemUUID:ec2285ee-83a7-9665-1629-da4634b12006 BootID:baeea569-2034-4db5-a8ed-1776dde9bb91 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:53:5e:06:cd:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:53:5e:06:cd:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:6a:ac:a6:a3:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 31 14:12:28.253568 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.253565 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 31 14:12:28.253684 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.253673 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 31 14:12:28.254780 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.254765 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 31 14:12:28.256871 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.256834 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 31 14:12:28.257019 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.256876 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-38.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 31 14:12:28.257517 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.257507 2577 topology_manager.go:138] "Creating topology manager with none policy" Jul 31 14:12:28.257549 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.257520 2577 container_manager_linux.go:309] "Creating device plugin manager" Jul 31 14:12:28.257549 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.257527 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 31 14:12:28.258861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.258851 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 31 14:12:28.259623 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.259614 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 31 14:12:28.262670 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.262658 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 31 14:12:28.262792 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.262782 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 31 14:12:28.262823 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.262814 2577 kubelet.go:486] "Attempting to sync node with API server" Jul 31 14:12:28.262866 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.262850 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 31 14:12:28.262902 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.262876 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 31 14:12:28.262902 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.262889 2577 kubelet.go:398] "Adding apiserver pod source" Jul 31 14:12:28.262902 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.262901 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 31 14:12:28.263766 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.263750 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 31 14:12:28.263766 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.263770 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 31 14:12:28.266765 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.266745 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 31 14:12:28.268082 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.268066 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 31 14:12:28.268160 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.268098 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 31 14:12:28.269162 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.269148 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 31 14:12:28.269162 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.269165 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 31 14:12:28.269290 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.269171 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 31 14:12:28.269290 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.269177 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 31 14:12:28.269290 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.269186 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 31 14:12:28.269290 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.269191 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 31 14:12:28.269290 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.269198 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 31 14:12:28.269290 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.269203 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 31 14:12:28.269290 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.269210 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 31 14:12:28.269290 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.269216 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 31 14:12:28.269290 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.269225 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 31 14:12:28.269290 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.269230 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 31 14:12:28.269962 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.269953 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 31 14:12:28.269962 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.269962 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 31 14:12:28.272940 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.272925 2577 server.go:1268] "Started kubelet" Jul 31 14:12:28.273044 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.273017 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 31 14:12:28.273151 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.273109 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 31 14:12:28.273211 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.273178 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 31 14:12:28.273756 ip-10-0-132-38 systemd[1]: Started Kubernetes Kubelet. Jul 31 14:12:28.275135 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.275118 2577 server.go:310] "Adding debug handlers to kubelet server" Jul 31 14:12:28.275320 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.275297 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 31 14:12:28.275675 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.275658 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-38.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 31 14:12:28.275809 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.275785 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 31 14:12:28.275911 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.275803 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-38.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 31 14:12:28.276646 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.276628 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vtvz2" Jul 31 14:12:28.278305 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.278288 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 31 14:12:28.280509 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.280492 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 31 14:12:28.280599 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.280512 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 31 14:12:28.280599 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.280562 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-38.ec2.internal\" not found" Jul 31 14:12:28.281063 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.281002 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 31 14:12:28.281063 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.281020 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 31 14:12:28.281221 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.281105 2577 reconstruct.go:97] "Volume reconstruction finished" Jul 31 14:12:28.281221 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.281114 2577 reconciler.go:29] "Reconciler: start to sync state" Jul 31 14:12:28.281221 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.281120 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 31 14:12:28.281664 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.281651 2577 factory.go:153] Registering CRI-O factory Jul 31 14:12:28.281745 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.281667 2577 factory.go:223] Registration of the crio container factory successfully Jul 31 14:12:28.281745 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.281717 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 31 14:12:28.281745 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.281726 2577 factory.go:55] Registering systemd factory Jul 31 14:12:28.281745 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.281733 2577 factory.go:223] Registration of the systemd container factory successfully Jul 31 14:12:28.281947 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.281754 2577 factory.go:103] Registering Raw factory Jul 31 14:12:28.281947 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.281768 2577 manager.go:1196] Started watching for new ooms in manager Jul 31 14:12:28.281947 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.280928 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-38.ec2.internal.18c7654a94ba2e54 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-38.ec2.internal,UID:ip-10-0-132-38.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-38.ec2.internal,},FirstTimestamp:2026-07-31 14:12:28.272905812 +0000 UTC m=+0.309728869,LastTimestamp:2026-07-31 14:12:28.272905812 +0000 UTC m=+0.309728869,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-38.ec2.internal,}" Jul 31 14:12:28.282208 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.282192 2577 manager.go:349] Starting recovery of all containers Jul 31 14:12:28.287039 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.287015 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vtvz2" Jul 31 14:12:28.288089 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.288057 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-38.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 31 14:12:28.288177 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.288111 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 31 14:12:28.293748 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.293726 2577 manager.go:354] Recovery completed Jul 31 14:12:28.297252 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.297232 2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 31 14:12:28.300197 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.300184 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:12:28.302775 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.302759 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:12:28.302869 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.302792 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:12:28.302869 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.302807 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:12:28.303366 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.303353 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 31 14:12:28.303366 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.303360 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:12:28.303456 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.303364 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 31 14:12:28.303456 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.303402 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 31 14:12:28.304069 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.304055 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:12:28.304128 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.304080 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:12:28.304128 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.304089 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:12:28.304565 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.304547 2577 policy_none.go:49] "None policy: Start" Jul 31 14:12:28.304565 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.304567 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 31 14:12:28.304698 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.304581 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 31 14:12:28.304876 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.304764 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-38.ec2.internal.18c7654a9681f5b7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-38.ec2.internal,UID:ip-10-0-132-38.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-38.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-38.ec2.internal,},FirstTimestamp:2026-07-31 14:12:28.302775735 +0000 UTC m=+0.339598800,LastTimestamp:2026-07-31 14:12:28.302775735 +0000 UTC m=+0.339598800,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-38.ec2.internal,}" Jul 31 14:12:28.305621 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.305611 2577 policy_none.go:47] "Start" Jul 31 14:12:28.347516 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.347381 2577 manager.go:337] "Starting Device Plugin manager" Jul 31 14:12:28.347626 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.347547 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 31 14:12:28.347626 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.347561 2577 server.go:87] "Starting device plugin registration server" Jul 31 14:12:28.347852 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.347828 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 31 14:12:28.347901 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.347862 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 31 14:12:28.347995 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.347979 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 31 14:12:28.348077 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.348060 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 31 14:12:28.348077 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.348078 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 31 14:12:28.348637 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.348611 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 31 14:12:28.348700 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.348660 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-38.ec2.internal\" not found" Jul 31 14:12:28.422534 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.422507 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 31 14:12:28.423688 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.423670 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 31 14:12:28.423746 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.423708 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 31 14:12:28.423746 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.423735 2577 kubelet.go:2448] "Starting kubelet main sync loop" Jul 31 14:12:28.423821 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.423766 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 31 14:12:28.426556 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.426535 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 14:12:28.448510 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.448497 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:12:28.449228 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.449214 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:12:28.449309 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.449242 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:12:28.449309 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.449252 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:12:28.449309 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.449274 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-38.ec2.internal" Jul 31 14:12:28.455507 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.455468 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-38.ec2.internal" Jul 31 14:12:28.455507 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.455488 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-38.ec2.internal\": node \"ip-10-0-132-38.ec2.internal\" not found" Jul 31 14:12:28.468348 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.468325 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-38.ec2.internal\" not found" Jul 31 14:12:28.523901 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.523874 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-38.ec2.internal"] Jul 31 14:12:28.523993 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.523937 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:12:28.524770 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.524753 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:12:28.524857 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.524786 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:12:28.524857 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.524801 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:12:28.526169 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.526154 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:12:28.526281 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.526265 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal" Jul 31 14:12:28.526281 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.526295 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:12:28.526841 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.526826 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:12:28.526932 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.526851 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:12:28.526932 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.526862 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:12:28.527013 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.526932 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:12:28.527013 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.526961 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:12:28.527013 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.526975 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:12:28.527964 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.527950 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-38.ec2.internal" Jul 31 14:12:28.528018 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.527976 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:12:28.528675 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.528658 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:12:28.528728 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.528693 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:12:28.528728 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.528707 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:12:28.543541 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.543521 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-38.ec2.internal\" not found" node="ip-10-0-132-38.ec2.internal" Jul 31 14:12:28.547152 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.547135 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-38.ec2.internal\" not found" node="ip-10-0-132-38.ec2.internal" Jul 31 14:12:28.568972 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.568950 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-38.ec2.internal\" not found" Jul 31 14:12:28.583047 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.583013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2b4caf14cfc48bdf71cf245b1d50495-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal\" (UID: \"f2b4caf14cfc48bdf71cf245b1d50495\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal" Jul 31 14:12:28.583139 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.583054 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/244611c7bbcc8364a26d08e8f81e9ee4-config\") pod \"kube-apiserver-proxy-ip-10-0-132-38.ec2.internal\" (UID: \"244611c7bbcc8364a26d08e8f81e9ee4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-38.ec2.internal" Jul 31 14:12:28.583139 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.583081 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2b4caf14cfc48bdf71cf245b1d50495-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal\" (UID: \"f2b4caf14cfc48bdf71cf245b1d50495\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal" Jul 31 14:12:28.670047 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.670019 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-38.ec2.internal\" not found" Jul 31 14:12:28.683318 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.683294 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2b4caf14cfc48bdf71cf245b1d50495-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal\" (UID: \"f2b4caf14cfc48bdf71cf245b1d50495\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal" Jul 31 14:12:28.683407 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.683336 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/244611c7bbcc8364a26d08e8f81e9ee4-config\") pod \"kube-apiserver-proxy-ip-10-0-132-38.ec2.internal\" (UID: \"244611c7bbcc8364a26d08e8f81e9ee4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-38.ec2.internal" Jul 31 14:12:28.683407 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.683363 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2b4caf14cfc48bdf71cf245b1d50495-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal\" (UID: \"f2b4caf14cfc48bdf71cf245b1d50495\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal" Jul 31 14:12:28.683407 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.683393 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2b4caf14cfc48bdf71cf245b1d50495-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal\" (UID: \"f2b4caf14cfc48bdf71cf245b1d50495\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal" Jul 31 14:12:28.683523 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.683429 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/244611c7bbcc8364a26d08e8f81e9ee4-config\") pod \"kube-apiserver-proxy-ip-10-0-132-38.ec2.internal\" (UID: \"244611c7bbcc8364a26d08e8f81e9ee4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-38.ec2.internal" Jul 31 14:12:28.683523 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.683469 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2b4caf14cfc48bdf71cf245b1d50495-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal\" (UID: \"f2b4caf14cfc48bdf71cf245b1d50495\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal" Jul 31 14:12:28.771024 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.770959 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-38.ec2.internal\" not found" Jul 31 14:12:28.850358 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.850322 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-38.ec2.internal" Jul 31 14:12:28.850444 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:28.850384 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal" Jul 31 14:12:28.871726 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.871697 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-38.ec2.internal\" not found" Jul 31 14:12:28.973906 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:28.973876 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-38.ec2.internal\" not found" Jul 31 14:12:29.075516 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:29.075455 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-38.ec2.internal\" not found" Jul 31 14:12:29.145701 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:29.145679 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 14:12:29.177076 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:29.177049 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-38.ec2.internal\" not found" Jul 31 14:12:29.225425 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:29.225403 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 31 14:12:29.225851 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:29.225555 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 31 14:12:29.225851 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:29.225561 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 31 14:12:29.244141 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:29.244124 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 14:12:29.277428 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:29.277396 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-38.ec2.internal\" not found" Jul 31 14:12:29.291394 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:29.291343 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 14:07:28 +0000 UTC" deadline="2028-05-08 17:49:27.584334926 +0000 UTC" Jul 31 14:12:29.291394 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:29.291381 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15531h36m58.292957017s" Jul 31 14:12:29.377828 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:29.377796 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-38.ec2.internal\" not found" Jul 31 14:12:29.420727 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:29.420685 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244611c7bbcc8364a26d08e8f81e9ee4.slice/crio-c4df618a42a938af4aa26284d4ddafdd1f8e4c9f8b4345eec43a54653f0f9e39 WatchSource:0}: Error finding container c4df618a42a938af4aa26284d4ddafdd1f8e4c9f8b4345eec43a54653f0f9e39: Status 404 returned error can't find the container with id c4df618a42a938af4aa26284d4ddafdd1f8e4c9f8b4345eec43a54653f0f9e39 Jul 31 14:12:29.421224 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:29.421201 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b4caf14cfc48bdf71cf245b1d50495.slice/crio-c1b84b4eb22ff3fd62484fb004b7c98f5756eb10fab96e1f23323f97d599bd59 WatchSource:0}: Error finding container c1b84b4eb22ff3fd62484fb004b7c98f5756eb10fab96e1f23323f97d599bd59: Status 404 returned error can't find the container with id c1b84b4eb22ff3fd62484fb004b7c98f5756eb10fab96e1f23323f97d599bd59 Jul 31 14:12:29.427733 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:29.427690 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal" event={"ID":"f2b4caf14cfc48bdf71cf245b1d50495","Type":"ContainerStarted","Data":"c1b84b4eb22ff3fd62484fb004b7c98f5756eb10fab96e1f23323f97d599bd59"} Jul 31 14:12:29.428010 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:29.427993 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 14:12:29.428925 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:29.428889 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-38.ec2.internal" event={"ID":"244611c7bbcc8364a26d08e8f81e9ee4","Type":"ContainerStarted","Data":"c4df618a42a938af4aa26284d4ddafdd1f8e4c9f8b4345eec43a54653f0f9e39"} Jul 31 14:12:29.480712 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:29.480691 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-38.ec2.internal\" not found" Jul 31 14:12:29.585307 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:29.585239 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-38.ec2.internal\" not found" Jul 31 14:12:29.648448 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:29.648421 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 14:12:29.680902 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:29.680875 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal" Jul 31 14:12:29.693064 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:29.693042 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 31 14:12:29.694050 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:29.694036 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-38.ec2.internal" Jul 31 14:12:29.699532 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:29.699508 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 31 14:12:30.264216 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.264178 2577 apiserver.go:52] "Watching apiserver" Jul 31 14:12:30.269915 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.269871 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 31 14:12:30.270354 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.270325 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-g9plr","openshift-multus/multus-wx97w","kube-system/global-pull-secret-syncer-lsrr4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w","openshift-cluster-node-tuning-operator/tuned-wzqr6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal","openshift-multus/network-metrics-daemon-skd9g","openshift-network-diagnostics/network-check-target-229tg","openshift-network-operator/iptables-alerter-htxt8","openshift-ovn-kubernetes/ovnkube-node-6sml2","kube-system/konnectivity-agent-hclhf","kube-system/kube-apiserver-proxy-ip-10-0-132-38.ec2.internal","openshift-dns/node-resolver-f2v62","openshift-image-registry/node-ca-qltd5"] Jul 31 14:12:30.272233 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.272203 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:30.272350 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:30.272310 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:12:30.273418 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.273395 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f2v62" Jul 31 14:12:30.274623 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.274602 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:30.274735 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:30.274702 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsrr4" podUID="72dad2d5-e7ca-4509-8619-e30eb609e348" Jul 31 14:12:30.275532 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.275506 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 31 14:12:30.275761 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.275743 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kv7hd\"" Jul 31 14:12:30.276045 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.276027 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.276641 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.276590 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 31 14:12:30.277326 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.277291 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.277943 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.277858 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 31 14:12:30.278042 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.277938 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 31 14:12:30.278042 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.277974 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fg966\"" Jul 31 14:12:30.278178 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.278154 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 31 14:12:30.279031 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.278958 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qltd5" Jul 31 14:12:30.279256 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.279236 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6rkgr\"" Jul 31 14:12:30.279537 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.279513 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 31 14:12:30.279719 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.279640 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 31 14:12:30.280756 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.280543 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-htxt8" Jul 31 14:12:30.281027 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.281007 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 31 14:12:30.281178 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.281167 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 31 14:12:30.281248 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.281180 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pxpfb\"" Jul 31 14:12:30.281694 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.281526 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 31 14:12:30.281694 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.281585 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 31 14:12:30.282346 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.282168 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 31 14:12:30.282346 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.282269 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.283074 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.282972 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 31 14:12:30.283255 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.283236 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 31 14:12:30.283531 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.283499 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jt8kf\"" Jul 31 14:12:30.284836 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.283656 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.284836 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.283954 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 31 14:12:30.284836 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.284180 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 31 14:12:30.284836 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.284304 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 31 14:12:30.284836 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.284666 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 31 14:12:30.284836 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.284706 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fk7r5\"" Jul 31 14:12:30.284836 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.284741 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 31 14:12:30.285890 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.285317 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sr4l2\"" Jul 31 14:12:30.285890 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.285481 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 31 14:12:30.286037 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.285899 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.287650 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.287608 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hclhf" Jul 31 14:12:30.288052 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.288031 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 31 14:12:30.288391 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.288372 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 31 14:12:30.289144 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.289115 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:30.289281 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:30.289255 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-229tg" podUID="0c82a3c9-4883-4803-9762-f008f3dfea21" Jul 31 14:12:30.290450 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.290420 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 31 14:12:30.290952 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.290896 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/37030756-473f-49b8-b47d-8e1e28df8150-sys-fs\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.291056 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.290973 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g4cs\" (UniqueName: \"kubernetes.io/projected/37030756-473f-49b8-b47d-8e1e28df8150-kube-api-access-7g4cs\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.291056 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.291016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a3824fc-d397-431c-8796-c494dc5d022b-system-cni-dir\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.291204 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.291185 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/37030756-473f-49b8-b47d-8e1e28df8150-socket-dir\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.291256 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.291225 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a3824fc-d397-431c-8796-c494dc5d022b-os-release\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.291313 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.291263 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-run\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.291313 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.291293 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 31 14:12:30.291674 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.291548 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 31 14:12:30.292862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.291769 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-646fl\"" Jul 31 14:12:30.292862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.291799 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 31 14:12:30.292862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.291298 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs\") pod \"network-metrics-daemon-skd9g\" (UID: \"78e66929-08a3-4aa1-8a96-740f8f8eb7c8\") " pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:30.292862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292209 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4qzh\" (UniqueName: \"kubernetes.io/projected/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-kube-api-access-m4qzh\") pod \"network-metrics-daemon-skd9g\" (UID: \"78e66929-08a3-4aa1-8a96-740f8f8eb7c8\") " pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:30.292862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292247 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/72dad2d5-e7ca-4509-8619-e30eb609e348-kubelet-config\") pod \"global-pull-secret-syncer-lsrr4\" (UID: \"72dad2d5-e7ca-4509-8619-e30eb609e348\") " pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:30.292862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/72dad2d5-e7ca-4509-8619-e30eb609e348-dbus\") pod \"global-pull-secret-syncer-lsrr4\" (UID: \"72dad2d5-e7ca-4509-8619-e30eb609e348\") " pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:30.292862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292320 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-modprobe-d\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.292862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292353 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gb59\" (UniqueName: \"kubernetes.io/projected/ea37acc1-4e8e-4884-ae89-48c6aa635f03-kube-api-access-2gb59\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.292862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292447 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/081e32e7-b1a0-49c3-87c6-1f690f0f09d7-iptables-alerter-script\") pod \"iptables-alerter-htxt8\" (UID: \"081e32e7-b1a0-49c3-87c6-1f690f0f09d7\") " pod="openshift-network-operator/iptables-alerter-htxt8" Jul 31 14:12:30.292862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292483 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/37030756-473f-49b8-b47d-8e1e28df8150-device-dir\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.292862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292518 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a3824fc-d397-431c-8796-c494dc5d022b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.292862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292533 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 31 14:12:30.292862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292553 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80dc0e3d-60fa-46e3-9e00-20eefb9986eb-tmp-dir\") pod \"node-resolver-f2v62\" (UID: \"80dc0e3d-60fa-46e3-9e00-20eefb9986eb\") " pod="openshift-dns/node-resolver-f2v62" Jul 31 14:12:30.292862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292582 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/37030756-473f-49b8-b47d-8e1e28df8150-registration-dir\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.292862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292622 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret\") pod \"global-pull-secret-syncer-lsrr4\" (UID: \"72dad2d5-e7ca-4509-8619-e30eb609e348\") " pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:30.292862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292655 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rghzl\" (UniqueName: \"kubernetes.io/projected/0fd2b5b1-f721-4a2e-8c19-1c220634d162-kube-api-access-rghzl\") pod \"node-ca-qltd5\" (UID: \"0fd2b5b1-f721-4a2e-8c19-1c220634d162\") " pod="openshift-image-registry/node-ca-qltd5" Jul 31 14:12:30.292862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292689 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/37030756-473f-49b8-b47d-8e1e28df8150-etc-selinux\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.293736 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a3824fc-d397-431c-8796-c494dc5d022b-cni-binary-copy\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.293736 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a3824fc-d397-431c-8796-c494dc5d022b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.293736 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292786 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a3824fc-d397-431c-8796-c494dc5d022b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.293736 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292808 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2srmt\"" Jul 31 14:12:30.293736 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292827 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbqzc\" (UniqueName: \"kubernetes.io/projected/80dc0e3d-60fa-46e3-9e00-20eefb9986eb-kube-api-access-jbqzc\") pod \"node-resolver-f2v62\" (UID: \"80dc0e3d-60fa-46e3-9e00-20eefb9986eb\") " pod="openshift-dns/node-resolver-f2v62" Jul 31 14:12:30.293736 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292937 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 31 14:12:30.293736 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-sysconfig\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.293736 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.292986 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-sysctl-d\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.293736 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.293042 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-sysctl-conf\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.293736 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.293083 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/80dc0e3d-60fa-46e3-9e00-20eefb9986eb-hosts-file\") pod \"node-resolver-f2v62\" (UID: \"80dc0e3d-60fa-46e3-9e00-20eefb9986eb\") " pod="openshift-dns/node-resolver-f2v62" Jul 31 14:12:30.293736 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.293118 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-kubernetes\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.293736 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.293149 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-systemd\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.293736 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.293214 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-sys\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.293736 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.293253 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-lib-modules\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.293736 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.293313 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fd2b5b1-f721-4a2e-8c19-1c220634d162-host\") pod \"node-ca-qltd5\" (UID: \"0fd2b5b1-f721-4a2e-8c19-1c220634d162\") " pod="openshift-image-registry/node-ca-qltd5" Jul 31 14:12:30.294469 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.293932 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0fd2b5b1-f721-4a2e-8c19-1c220634d162-serviceca\") pod \"node-ca-qltd5\" (UID: \"0fd2b5b1-f721-4a2e-8c19-1c220634d162\") " pod="openshift-image-registry/node-ca-qltd5" Jul 31 14:12:30.294469 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.293967 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/081e32e7-b1a0-49c3-87c6-1f690f0f09d7-host-slash\") pod \"iptables-alerter-htxt8\" (UID: \"081e32e7-b1a0-49c3-87c6-1f690f0f09d7\") " pod="openshift-network-operator/iptables-alerter-htxt8" Jul 31 14:12:30.294469 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.294039 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/37030756-473f-49b8-b47d-8e1e28df8150-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.294469 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.294080 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-var-lib-kubelet\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.294469 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.294116 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-tuned\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.294469 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.294299 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea37acc1-4e8e-4884-ae89-48c6aa635f03-tmp\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.294469 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.294408 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s8kg\" (UniqueName: \"kubernetes.io/projected/081e32e7-b1a0-49c3-87c6-1f690f0f09d7-kube-api-access-4s8kg\") pod \"iptables-alerter-htxt8\" (UID: \"081e32e7-b1a0-49c3-87c6-1f690f0f09d7\") " pod="openshift-network-operator/iptables-alerter-htxt8" Jul 31 14:12:30.294469 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.294458 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a3824fc-d397-431c-8796-c494dc5d022b-cnibin\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.294844 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.294547 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjlx\" (UniqueName: \"kubernetes.io/projected/1a3824fc-d397-431c-8796-c494dc5d022b-kube-api-access-6xjlx\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.294844 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.294634 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-host\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.295697 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.295674 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 31 14:12:30.317397 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.317371 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4d7nd" Jul 31 14:12:30.326428 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.326389 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4d7nd" Jul 31 14:12:30.383043 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.383020 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 31 14:12:30.395600 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.395570 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6a540518-e608-4021-a467-eaa910669373-konnectivity-ca\") pod \"konnectivity-agent-hclhf\" (UID: \"6a540518-e608-4021-a467-eaa910669373\") " pod="kube-system/konnectivity-agent-hclhf" Jul 31 14:12:30.395722 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.395626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/37030756-473f-49b8-b47d-8e1e28df8150-sys-fs\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.395722 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.395656 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a3824fc-d397-431c-8796-c494dc5d022b-system-cni-dir\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.395722 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.395683 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/37030756-473f-49b8-b47d-8e1e28df8150-socket-dir\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.395722 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.395693 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/37030756-473f-49b8-b47d-8e1e28df8150-sys-fs\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.395722 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.395710 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a3824fc-d397-431c-8796-c494dc5d022b-os-release\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.396057 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.395749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gb59\" (UniqueName: \"kubernetes.io/projected/ea37acc1-4e8e-4884-ae89-48c6aa635f03-kube-api-access-2gb59\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.396057 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.395750 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a3824fc-d397-431c-8796-c494dc5d022b-system-cni-dir\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.396057 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.395770 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs\") pod \"network-metrics-daemon-skd9g\" (UID: \"78e66929-08a3-4aa1-8a96-740f8f8eb7c8\") " pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:30.396057 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.395786 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a3824fc-d397-431c-8796-c494dc5d022b-os-release\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.396057 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.395789 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4qzh\" (UniqueName: \"kubernetes.io/projected/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-kube-api-access-m4qzh\") pod \"network-metrics-daemon-skd9g\" (UID: \"78e66929-08a3-4aa1-8a96-740f8f8eb7c8\") " pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:30.396057 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.395829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/72dad2d5-e7ca-4509-8619-e30eb609e348-kubelet-config\") pod \"global-pull-secret-syncer-lsrr4\" (UID: \"72dad2d5-e7ca-4509-8619-e30eb609e348\") " pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:30.396057 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.395854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/72dad2d5-e7ca-4509-8619-e30eb609e348-dbus\") pod \"global-pull-secret-syncer-lsrr4\" (UID: \"72dad2d5-e7ca-4509-8619-e30eb609e348\") " pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:30.396057 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.395905 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-var-lib-openvswitch\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.396057 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.395909 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/37030756-473f-49b8-b47d-8e1e28df8150-socket-dir\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.396057 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.395935 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-run-openvswitch\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.396057 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.395962 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d3c0ddd-40ab-42dd-ba02-186210158dcf-ovnkube-script-lib\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.396057 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.395988 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/37030756-473f-49b8-b47d-8e1e28df8150-device-dir\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.396057 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:30.396005 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:30.396057 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-cnibin\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.396057 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396040 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret\") pod \"global-pull-secret-syncer-lsrr4\" (UID: \"72dad2d5-e7ca-4509-8619-e30eb609e348\") " pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:30.396057 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396046 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/72dad2d5-e7ca-4509-8619-e30eb609e348-kubelet-config\") pod \"global-pull-secret-syncer-lsrr4\" (UID: \"72dad2d5-e7ca-4509-8619-e30eb609e348\") " pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:30.396875 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:30.396072 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs podName:78e66929-08a3-4aa1-8a96-740f8f8eb7c8 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:30.896049634 +0000 UTC m=+2.932872706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs") pod "network-metrics-daemon-skd9g" (UID: "78e66929-08a3-4aa1-8a96-740f8f8eb7c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:30.396875 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-host-run-k8s-cni-cncf-io\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.396875 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396119 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/37030756-473f-49b8-b47d-8e1e28df8150-device-dir\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.396875 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396123 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-host-var-lib-cni-bin\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.396875 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-hostroot\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.396875 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396169 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c3047cea-d168-4307-a588-6c1f60e9a921-multus-daemon-config\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.396875 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:30.396189 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:30.396875 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396195 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-cni-netd\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.396875 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396195 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/72dad2d5-e7ca-4509-8619-e30eb609e348-dbus\") pod \"global-pull-secret-syncer-lsrr4\" (UID: \"72dad2d5-e7ca-4509-8619-e30eb609e348\") " pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:30.396875 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396220 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-os-release\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.396875 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:30.396247 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret podName:72dad2d5-e7ca-4509-8619-e30eb609e348 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:30.89623024 +0000 UTC m=+2.933053303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret") pod "global-pull-secret-syncer-lsrr4" (UID: "72dad2d5-e7ca-4509-8619-e30eb609e348") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:30.396875 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396334 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-host-var-lib-cni-multus\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.396875 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396376 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a3824fc-d397-431c-8796-c494dc5d022b-cni-binary-copy\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.396875 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-sysconfig\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.396875 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396431 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-sysctl-d\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.396875 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396458 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-sysctl-conf\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.397695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-systemd\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.397695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-sys\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.397695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396556 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-sysconfig\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.397695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/80dc0e3d-60fa-46e3-9e00-20eefb9986eb-hosts-file\") pod \"node-resolver-f2v62\" (UID: \"80dc0e3d-60fa-46e3-9e00-20eefb9986eb\") " pod="openshift-dns/node-resolver-f2v62" Jul 31 14:12:30.397695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-lib-modules\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.397695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396615 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fd2b5b1-f721-4a2e-8c19-1c220634d162-host\") pod \"node-ca-qltd5\" (UID: \"0fd2b5b1-f721-4a2e-8c19-1c220634d162\") " pod="openshift-image-registry/node-ca-qltd5" Jul 31 14:12:30.397695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396638 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-sysctl-d\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.397695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396642 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-host-run-multus-certs\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.397695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396682 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d3c0ddd-40ab-42dd-ba02-186210158dcf-env-overrides\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.397695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396719 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-tuned\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.397695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396737 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-sysctl-conf\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.397695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396745 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-multus-cni-dir\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.397695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396805 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fd2b5b1-f721-4a2e-8c19-1c220634d162-host\") pod \"node-ca-qltd5\" (UID: \"0fd2b5b1-f721-4a2e-8c19-1c220634d162\") " pod="openshift-image-registry/node-ca-qltd5" Jul 31 14:12:30.397695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396822 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-lib-modules\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.397695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396853 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3047cea-d168-4307-a588-6c1f60e9a921-cni-binary-copy\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.397695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396883 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-multus-socket-dir-parent\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.397695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396926 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.398585 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396968 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wl79\" (UniqueName: \"kubernetes.io/projected/2d3c0ddd-40ab-42dd-ba02-186210158dcf-kube-api-access-8wl79\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.398585 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396977 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/80dc0e3d-60fa-46e3-9e00-20eefb9986eb-hosts-file\") pod \"node-resolver-f2v62\" (UID: \"80dc0e3d-60fa-46e3-9e00-20eefb9986eb\") " pod="openshift-dns/node-resolver-f2v62" Jul 31 14:12:30.398585 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.396994 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-systemd\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.398585 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397010 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a3824fc-d397-431c-8796-c494dc5d022b-cnibin\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.398585 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397072 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a3824fc-d397-431c-8796-c494dc5d022b-cni-binary-copy\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.398585 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397080 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-sys\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.398585 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397060 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-host-run-netns\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.398585 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397095 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a3824fc-d397-431c-8796-c494dc5d022b-cnibin\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.398585 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397163 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-etc-openvswitch\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.398585 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397163 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 31 14:12:30.398585 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g4cs\" (UniqueName: \"kubernetes.io/projected/37030756-473f-49b8-b47d-8e1e28df8150-kube-api-access-7g4cs\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.398585 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397277 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-run\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.398585 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397301 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-kubelet\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.398585 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-modprobe-d\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.398585 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397359 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-slash\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.398585 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397386 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6a540518-e608-4021-a467-eaa910669373-agent-certs\") pod \"konnectivity-agent-hclhf\" (UID: \"6a540518-e608-4021-a467-eaa910669373\") " pod="kube-system/konnectivity-agent-hclhf" Jul 31 14:12:30.398585 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397415 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/081e32e7-b1a0-49c3-87c6-1f690f0f09d7-iptables-alerter-script\") pod \"iptables-alerter-htxt8\" (UID: \"081e32e7-b1a0-49c3-87c6-1f690f0f09d7\") " pod="openshift-network-operator/iptables-alerter-htxt8" Jul 31 14:12:30.399466 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397456 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-modprobe-d\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.399466 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a3824fc-d397-431c-8796-c494dc5d022b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.399466 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397506 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80dc0e3d-60fa-46e3-9e00-20eefb9986eb-tmp-dir\") pod \"node-resolver-f2v62\" (UID: \"80dc0e3d-60fa-46e3-9e00-20eefb9986eb\") " pod="openshift-dns/node-resolver-f2v62" Jul 31 14:12:30.399466 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397585 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-host-var-lib-kubelet\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.399466 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397636 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-systemd-units\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.399466 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397665 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-run-ovn\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.399466 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397758 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-cni-bin\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.399466 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397780 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d3c0ddd-40ab-42dd-ba02-186210158dcf-ovn-node-metrics-cert\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.399466 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397799 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/37030756-473f-49b8-b47d-8e1e28df8150-registration-dir\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.399466 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397816 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rghzl\" (UniqueName: \"kubernetes.io/projected/0fd2b5b1-f721-4a2e-8c19-1c220634d162-kube-api-access-rghzl\") pod \"node-ca-qltd5\" (UID: \"0fd2b5b1-f721-4a2e-8c19-1c220634d162\") " pod="openshift-image-registry/node-ca-qltd5" Jul 31 14:12:30.399466 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397831 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-run-netns\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.399466 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d3c0ddd-40ab-42dd-ba02-186210158dcf-ovnkube-config\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.399466 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397866 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shtps\" (UniqueName: \"kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps\") pod \"network-check-target-229tg\" (UID: \"0c82a3c9-4883-4803-9762-f008f3dfea21\") " pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:30.399466 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397924 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/37030756-473f-49b8-b47d-8e1e28df8150-etc-selinux\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.399466 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.397937 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a3824fc-d397-431c-8796-c494dc5d022b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.399466 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398000 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a3824fc-d397-431c-8796-c494dc5d022b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.400287 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398028 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a3824fc-d397-431c-8796-c494dc5d022b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.400287 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398046 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80dc0e3d-60fa-46e3-9e00-20eefb9986eb-tmp-dir\") pod \"node-resolver-f2v62\" (UID: \"80dc0e3d-60fa-46e3-9e00-20eefb9986eb\") " pod="openshift-dns/node-resolver-f2v62" Jul 31 14:12:30.400287 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398057 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbqzc\" (UniqueName: \"kubernetes.io/projected/80dc0e3d-60fa-46e3-9e00-20eefb9986eb-kube-api-access-jbqzc\") pod \"node-resolver-f2v62\" (UID: \"80dc0e3d-60fa-46e3-9e00-20eefb9986eb\") " pod="openshift-dns/node-resolver-f2v62" Jul 31 14:12:30.400287 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398085 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwlxj\" (UniqueName: \"kubernetes.io/projected/c3047cea-d168-4307-a588-6c1f60e9a921-kube-api-access-xwlxj\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.400287 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398112 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-log-socket\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.400287 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398140 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.400287 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398172 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-kubernetes\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.400287 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398199 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0fd2b5b1-f721-4a2e-8c19-1c220634d162-serviceca\") pod \"node-ca-qltd5\" (UID: \"0fd2b5b1-f721-4a2e-8c19-1c220634d162\") " pod="openshift-image-registry/node-ca-qltd5" Jul 31 14:12:30.400287 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398236 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-system-cni-dir\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.400287 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/081e32e7-b1a0-49c3-87c6-1f690f0f09d7-host-slash\") pod \"iptables-alerter-htxt8\" (UID: \"081e32e7-b1a0-49c3-87c6-1f690f0f09d7\") " pod="openshift-network-operator/iptables-alerter-htxt8" Jul 31 14:12:30.400287 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398284 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/37030756-473f-49b8-b47d-8e1e28df8150-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.400287 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-var-lib-kubelet\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.400287 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398392 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/081e32e7-b1a0-49c3-87c6-1f690f0f09d7-iptables-alerter-script\") pod \"iptables-alerter-htxt8\" (UID: \"081e32e7-b1a0-49c3-87c6-1f690f0f09d7\") " pod="openshift-network-operator/iptables-alerter-htxt8" Jul 31 14:12:30.400287 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398398 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea37acc1-4e8e-4884-ae89-48c6aa635f03-tmp\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.400287 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-multus-conf-dir\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.400287 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398487 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-etc-kubernetes\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.400287 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398489 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a3824fc-d397-431c-8796-c494dc5d022b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.401209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398513 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-run-systemd\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.401209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398539 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-node-log\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.401209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s8kg\" (UniqueName: \"kubernetes.io/projected/081e32e7-b1a0-49c3-87c6-1f690f0f09d7-kube-api-access-4s8kg\") pod \"iptables-alerter-htxt8\" (UID: \"081e32e7-b1a0-49c3-87c6-1f690f0f09d7\") " pod="openshift-network-operator/iptables-alerter-htxt8" Jul 31 14:12:30.401209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398583 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/37030756-473f-49b8-b47d-8e1e28df8150-etc-selinux\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.401209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398597 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjlx\" (UniqueName: \"kubernetes.io/projected/1a3824fc-d397-431c-8796-c494dc5d022b-kube-api-access-6xjlx\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.401209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-host\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.401209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398650 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-run\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.401209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398702 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/37030756-473f-49b8-b47d-8e1e28df8150-registration-dir\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.401209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398750 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-kubernetes\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.401209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.399183 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a3824fc-d397-431c-8796-c494dc5d022b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.401209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.399191 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/081e32e7-b1a0-49c3-87c6-1f690f0f09d7-host-slash\") pod \"iptables-alerter-htxt8\" (UID: \"081e32e7-b1a0-49c3-87c6-1f690f0f09d7\") " pod="openshift-network-operator/iptables-alerter-htxt8" Jul 31 14:12:30.401209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.399193 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0fd2b5b1-f721-4a2e-8c19-1c220634d162-serviceca\") pod \"node-ca-qltd5\" (UID: \"0fd2b5b1-f721-4a2e-8c19-1c220634d162\") " pod="openshift-image-registry/node-ca-qltd5" Jul 31 14:12:30.401209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.398704 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-host\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.401209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.399242 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea37acc1-4e8e-4884-ae89-48c6aa635f03-var-lib-kubelet\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.401209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.399291 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/37030756-473f-49b8-b47d-8e1e28df8150-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.401998 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.401222 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ea37acc1-4e8e-4884-ae89-48c6aa635f03-etc-tuned\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.402884 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.402863 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea37acc1-4e8e-4884-ae89-48c6aa635f03-tmp\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.407985 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.407961 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gb59\" (UniqueName: \"kubernetes.io/projected/ea37acc1-4e8e-4884-ae89-48c6aa635f03-kube-api-access-2gb59\") pod \"tuned-wzqr6\" (UID: \"ea37acc1-4e8e-4884-ae89-48c6aa635f03\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.408162 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.408136 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4qzh\" (UniqueName: \"kubernetes.io/projected/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-kube-api-access-m4qzh\") pod \"network-metrics-daemon-skd9g\" (UID: \"78e66929-08a3-4aa1-8a96-740f8f8eb7c8\") " pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:30.408706 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.408682 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjlx\" (UniqueName: \"kubernetes.io/projected/1a3824fc-d397-431c-8796-c494dc5d022b-kube-api-access-6xjlx\") pod \"multus-additional-cni-plugins-g9plr\" (UID: \"1a3824fc-d397-431c-8796-c494dc5d022b\") " pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.409826 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.409777 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g4cs\" (UniqueName: \"kubernetes.io/projected/37030756-473f-49b8-b47d-8e1e28df8150-kube-api-access-7g4cs\") pod \"aws-ebs-csi-driver-node-tmw4w\" (UID: \"37030756-473f-49b8-b47d-8e1e28df8150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.410428 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.410401 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbqzc\" (UniqueName: \"kubernetes.io/projected/80dc0e3d-60fa-46e3-9e00-20eefb9986eb-kube-api-access-jbqzc\") pod \"node-resolver-f2v62\" (UID: \"80dc0e3d-60fa-46e3-9e00-20eefb9986eb\") " pod="openshift-dns/node-resolver-f2v62" Jul 31 14:12:30.410998 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.410974 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rghzl\" (UniqueName: \"kubernetes.io/projected/0fd2b5b1-f721-4a2e-8c19-1c220634d162-kube-api-access-rghzl\") pod \"node-ca-qltd5\" (UID: \"0fd2b5b1-f721-4a2e-8c19-1c220634d162\") " pod="openshift-image-registry/node-ca-qltd5" Jul 31 14:12:30.411486 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.411467 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s8kg\" (UniqueName: \"kubernetes.io/projected/081e32e7-b1a0-49c3-87c6-1f690f0f09d7-kube-api-access-4s8kg\") pod \"iptables-alerter-htxt8\" (UID: \"081e32e7-b1a0-49c3-87c6-1f690f0f09d7\") " pod="openshift-network-operator/iptables-alerter-htxt8" Jul 31 14:12:30.428170 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.428141 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 14:12:30.499372 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499334 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6a540518-e608-4021-a467-eaa910669373-agent-certs\") pod \"konnectivity-agent-hclhf\" (UID: \"6a540518-e608-4021-a467-eaa910669373\") " pod="kube-system/konnectivity-agent-hclhf" Jul 31 14:12:30.499558 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499382 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-host-var-lib-kubelet\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.499558 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499413 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-systemd-units\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.499558 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-run-ovn\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.499558 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499465 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-cni-bin\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.499558 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499493 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d3c0ddd-40ab-42dd-ba02-186210158dcf-ovn-node-metrics-cert\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.499558 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499498 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-host-var-lib-kubelet\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.499558 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499519 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-run-netns\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.499558 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499552 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d3c0ddd-40ab-42dd-ba02-186210158dcf-ovnkube-config\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.500014 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499557 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-run-ovn\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.500014 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499574 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-run-netns\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.500014 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499582 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtps\" (UniqueName: \"kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps\") pod \"network-check-target-229tg\" (UID: \"0c82a3c9-4883-4803-9762-f008f3dfea21\") " pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:30.500014 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499508 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-systemd-units\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.500014 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499615 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwlxj\" (UniqueName: \"kubernetes.io/projected/c3047cea-d168-4307-a588-6c1f60e9a921-kube-api-access-xwlxj\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.500014 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499616 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-cni-bin\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.500014 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499689 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-log-socket\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.500014 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499751 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-log-socket\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.500014 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499786 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.500014 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499824 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-system-cni-dir\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.500014 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499870 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-multus-conf-dir\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.500014 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-etc-kubernetes\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.500014 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499900 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.500014 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-run-systemd\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.500014 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499955 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-multus-conf-dir\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.500014 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.499997 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-node-log\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.500834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500030 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6a540518-e608-4021-a467-eaa910669373-konnectivity-ca\") pod \"konnectivity-agent-hclhf\" (UID: \"6a540518-e608-4021-a467-eaa910669373\") " pod="kube-system/konnectivity-agent-hclhf" Jul 31 14:12:30.500834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-var-lib-openvswitch\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.500834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500090 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-etc-kubernetes\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.500834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500114 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-run-openvswitch\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.500834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d3c0ddd-40ab-42dd-ba02-186210158dcf-ovnkube-script-lib\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.500834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-cnibin\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.500834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500358 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-system-cni-dir\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.500834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500399 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-node-log\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.500834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500436 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-run-systemd\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.500834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500486 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-cnibin\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.500834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-host-run-k8s-cni-cncf-io\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.500834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500568 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-host-var-lib-cni-bin\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.500834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500616 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-hostroot\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.500834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500632 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d3c0ddd-40ab-42dd-ba02-186210158dcf-ovnkube-config\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.500834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c3047cea-d168-4307-a588-6c1f60e9a921-multus-daemon-config\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.500834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500632 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6a540518-e608-4021-a467-eaa910669373-konnectivity-ca\") pod \"konnectivity-agent-hclhf\" (UID: \"6a540518-e608-4021-a467-eaa910669373\") " pod="kube-system/konnectivity-agent-hclhf" Jul 31 14:12:30.500834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500671 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-cni-netd\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.500834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500674 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-var-lib-openvswitch\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.501744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500694 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-run-openvswitch\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.501744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500732 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-host-var-lib-cni-bin\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.501744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500728 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d3c0ddd-40ab-42dd-ba02-186210158dcf-ovnkube-script-lib\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.501744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500743 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-os-release\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.501744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500788 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-cni-netd\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.501744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500791 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-hostroot\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.501744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500805 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-host-var-lib-cni-multus\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.501744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500897 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-host-var-lib-cni-multus\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.501744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500934 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-host-run-k8s-cni-cncf-io\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.501744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500948 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-host-run-multus-certs\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.501744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.500878 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-os-release\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.501744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501040 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d3c0ddd-40ab-42dd-ba02-186210158dcf-env-overrides\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.501744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-multus-cni-dir\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.501744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501105 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-host-run-multus-certs\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.501744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3047cea-d168-4307-a588-6c1f60e9a921-cni-binary-copy\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.501744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501180 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-multus-cni-dir\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.501744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-multus-socket-dir-parent\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.501744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501230 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c3047cea-d168-4307-a588-6c1f60e9a921-multus-daemon-config\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.502740 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501233 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.502740 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wl79\" (UniqueName: \"kubernetes.io/projected/2d3c0ddd-40ab-42dd-ba02-186210158dcf-kube-api-access-8wl79\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.502740 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501288 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-host-run-netns\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.502740 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501298 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-multus-socket-dir-parent\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.502740 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-etc-openvswitch\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.502740 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501326 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.502740 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501344 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-kubelet\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.502740 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501387 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3047cea-d168-4307-a588-6c1f60e9a921-host-run-netns\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.502740 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501395 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-kubelet\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.502740 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501420 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-slash\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.502740 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501440 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-etc-openvswitch\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.502740 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501481 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d3c0ddd-40ab-42dd-ba02-186210158dcf-host-slash\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.502740 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501488 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d3c0ddd-40ab-42dd-ba02-186210158dcf-env-overrides\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.502740 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.501608 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3047cea-d168-4307-a588-6c1f60e9a921-cni-binary-copy\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.502740 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.502710 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d3c0ddd-40ab-42dd-ba02-186210158dcf-ovn-node-metrics-cert\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.503755 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.503734 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6a540518-e608-4021-a467-eaa910669373-agent-certs\") pod \"konnectivity-agent-hclhf\" (UID: \"6a540518-e608-4021-a467-eaa910669373\") " pod="kube-system/konnectivity-agent-hclhf" Jul 31 14:12:30.505540 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:30.505490 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:12:30.505540 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:30.505517 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:12:30.505540 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:30.505534 2577 projected.go:196] Error preparing data for projected volume kube-api-access-shtps for pod openshift-network-diagnostics/network-check-target-229tg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:30.505742 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:30.505656 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps podName:0c82a3c9-4883-4803-9762-f008f3dfea21 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:31.005594827 +0000 UTC m=+3.042417891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-shtps" (UniqueName: "kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps") pod "network-check-target-229tg" (UID: "0c82a3c9-4883-4803-9762-f008f3dfea21") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:30.508862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.508831 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wl79\" (UniqueName: \"kubernetes.io/projected/2d3c0ddd-40ab-42dd-ba02-186210158dcf-kube-api-access-8wl79\") pod \"ovnkube-node-6sml2\" (UID: \"2d3c0ddd-40ab-42dd-ba02-186210158dcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.509388 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.509233 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwlxj\" (UniqueName: \"kubernetes.io/projected/c3047cea-d168-4307-a588-6c1f60e9a921-kube-api-access-xwlxj\") pod \"multus-wx97w\" (UID: \"c3047cea-d168-4307-a588-6c1f60e9a921\") " pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.587008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.586921 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f2v62" Jul 31 14:12:30.602164 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.602140 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" Jul 31 14:12:30.611988 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.611966 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" Jul 31 14:12:30.626660 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.626634 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qltd5" Jul 31 14:12:30.636089 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.636064 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-htxt8" Jul 31 14:12:30.644927 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.644904 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g9plr" Jul 31 14:12:30.651763 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.651740 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wx97w" Jul 31 14:12:30.659578 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.659557 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:30.667211 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.667188 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hclhf" Jul 31 14:12:30.906269 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.906173 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs\") pod \"network-metrics-daemon-skd9g\" (UID: \"78e66929-08a3-4aa1-8a96-740f8f8eb7c8\") " pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:30.906269 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:30.906224 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret\") pod \"global-pull-secret-syncer-lsrr4\" (UID: \"72dad2d5-e7ca-4509-8619-e30eb609e348\") " pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:30.906517 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:30.906356 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:30.906517 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:30.906370 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:30.906517 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:30.906428 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret podName:72dad2d5-e7ca-4509-8619-e30eb609e348 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:31.906408679 +0000 UTC m=+3.943231740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret") pod "global-pull-secret-syncer-lsrr4" (UID: "72dad2d5-e7ca-4509-8619-e30eb609e348") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:30.906517 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:30.906444 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs podName:78e66929-08a3-4aa1-8a96-740f8f8eb7c8 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:31.906437623 +0000 UTC m=+3.943260674 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs") pod "network-metrics-daemon-skd9g" (UID: "78e66929-08a3-4aa1-8a96-740f8f8eb7c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:31.007386 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:31.007343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtps\" (UniqueName: \"kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps\") pod \"network-check-target-229tg\" (UID: \"0c82a3c9-4883-4803-9762-f008f3dfea21\") " pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:31.007578 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:31.007542 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:12:31.007578 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:31.007566 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:12:31.007697 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:31.007583 2577 projected.go:196] Error preparing data for projected volume kube-api-access-shtps for pod openshift-network-diagnostics/network-check-target-229tg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:31.007697 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:31.007649 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps podName:0c82a3c9-4883-4803-9762-f008f3dfea21 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:32.007627532 +0000 UTC m=+4.044450603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-shtps" (UniqueName: "kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps") pod "network-check-target-229tg" (UID: "0c82a3c9-4883-4803-9762-f008f3dfea21") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:31.328794 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:31.328704 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:07:30 +0000 UTC" deadline="2028-05-03 06:43:30.887688488 +0000 UTC" Jul 31 14:12:31.328794 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:31.328745 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15400h30m59.558955681s" Jul 31 14:12:31.424035 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:31.423998 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:31.424222 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:31.424138 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-229tg" podUID="0c82a3c9-4883-4803-9762-f008f3dfea21" Jul 31 14:12:31.424292 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:31.424218 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:31.424385 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:31.424344 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsrr4" podUID="72dad2d5-e7ca-4509-8619-e30eb609e348" Jul 31 14:12:31.489237 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:31.489199 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081e32e7_b1a0_49c3_87c6_1f690f0f09d7.slice/crio-9f0d9f8753e32bc7b94ff369579a105e201217e37ab3cd6127fa1d1fae207848 WatchSource:0}: Error finding container 9f0d9f8753e32bc7b94ff369579a105e201217e37ab3cd6127fa1d1fae207848: Status 404 returned error can't find the container with id 9f0d9f8753e32bc7b94ff369579a105e201217e37ab3cd6127fa1d1fae207848 Jul 31 14:12:31.490798 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:31.490638 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37030756_473f_49b8_b47d_8e1e28df8150.slice/crio-78fa5cd91cf14bcd0f0e07ad0f99fba3ec03fca599d902c72469492dd1ba7cc8 WatchSource:0}: Error finding container 78fa5cd91cf14bcd0f0e07ad0f99fba3ec03fca599d902c72469492dd1ba7cc8: Status 404 returned error can't find the container with id 78fa5cd91cf14bcd0f0e07ad0f99fba3ec03fca599d902c72469492dd1ba7cc8 Jul 31 14:12:31.491477 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:31.491419 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80dc0e3d_60fa_46e3_9e00_20eefb9986eb.slice/crio-52361fef0886eeb81ece79415991a34dac21e75024c0ea66d50e58c6a45a02d0 WatchSource:0}: Error finding container 52361fef0886eeb81ece79415991a34dac21e75024c0ea66d50e58c6a45a02d0: Status 404 returned error can't find the container with id 52361fef0886eeb81ece79415991a34dac21e75024c0ea66d50e58c6a45a02d0 Jul 31 14:12:31.493620 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:31.493444 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea37acc1_4e8e_4884_ae89_48c6aa635f03.slice/crio-fdb6acce35e5186ec2f87b31b7eb821bae8d70fcd1ba515259630c49cfd519c7 WatchSource:0}: Error finding container fdb6acce35e5186ec2f87b31b7eb821bae8d70fcd1ba515259630c49cfd519c7: Status 404 returned error can't find the container with id fdb6acce35e5186ec2f87b31b7eb821bae8d70fcd1ba515259630c49cfd519c7 Jul 31 14:12:31.498004 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:31.497977 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3047cea_d168_4307_a588_6c1f60e9a921.slice/crio-1f65f78627d0960e5458a49b65f9622b66f163b3e41c38fb1a75d539cd0722ca WatchSource:0}: Error finding container 1f65f78627d0960e5458a49b65f9622b66f163b3e41c38fb1a75d539cd0722ca: Status 404 returned error can't find the container with id 1f65f78627d0960e5458a49b65f9622b66f163b3e41c38fb1a75d539cd0722ca Jul 31 14:12:31.498771 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:31.498724 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3c0ddd_40ab_42dd_ba02_186210158dcf.slice/crio-bacc5d075c963d71f26c5568a2cb8c3ea4acc27c0b4f2e090bb34c404bf090dc WatchSource:0}: Error finding container bacc5d075c963d71f26c5568a2cb8c3ea4acc27c0b4f2e090bb34c404bf090dc: Status 404 returned error can't find the container with id bacc5d075c963d71f26c5568a2cb8c3ea4acc27c0b4f2e090bb34c404bf090dc Jul 31 14:12:31.502400 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:12:31.501144 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd2b5b1_f721_4a2e_8c19_1c220634d162.slice/crio-acb41868124c5c71c808a4797f794d97ddd1c366e7c04fa875fc65f180343650 WatchSource:0}: Error finding container acb41868124c5c71c808a4797f794d97ddd1c366e7c04fa875fc65f180343650: Status 404 returned error can't find the container with id acb41868124c5c71c808a4797f794d97ddd1c366e7c04fa875fc65f180343650 Jul 31 14:12:31.615339 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:31.615312 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 14:12:31.915404 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:31.915150 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs\") pod \"network-metrics-daemon-skd9g\" (UID: \"78e66929-08a3-4aa1-8a96-740f8f8eb7c8\") " pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:31.915404 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:31.915330 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:31.915404 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:31.915371 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret\") pod \"global-pull-secret-syncer-lsrr4\" (UID: \"72dad2d5-e7ca-4509-8619-e30eb609e348\") " pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:31.915606 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:31.915408 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs podName:78e66929-08a3-4aa1-8a96-740f8f8eb7c8 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:33.915385823 +0000 UTC m=+5.952208869 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs") pod "network-metrics-daemon-skd9g" (UID: "78e66929-08a3-4aa1-8a96-740f8f8eb7c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:31.915606 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:31.915458 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:31.915606 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:31.915509 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret podName:72dad2d5-e7ca-4509-8619-e30eb609e348 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:33.915494819 +0000 UTC m=+5.952317884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret") pod "global-pull-secret-syncer-lsrr4" (UID: "72dad2d5-e7ca-4509-8619-e30eb609e348") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:32.016158 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:32.016117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtps\" (UniqueName: \"kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps\") pod \"network-check-target-229tg\" (UID: \"0c82a3c9-4883-4803-9762-f008f3dfea21\") " pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:32.016349 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:32.016329 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:12:32.016414 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:32.016355 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:12:32.016414 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:32.016369 2577 projected.go:196] Error preparing data for projected volume kube-api-access-shtps for pod openshift-network-diagnostics/network-check-target-229tg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:32.016533 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:32.016432 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps podName:0c82a3c9-4883-4803-9762-f008f3dfea21 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:34.016413286 +0000 UTC m=+6.053236351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-shtps" (UniqueName: "kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps") pod "network-check-target-229tg" (UID: "0c82a3c9-4883-4803-9762-f008f3dfea21") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:32.329525 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:32.329382 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:07:30 +0000 UTC" deadline="2027-12-31 14:21:37.428862937 +0000 UTC" Jul 31 14:12:32.329525 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:32.329447 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12432h9m5.09942063s" Jul 31 14:12:32.425625 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:32.425069 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:32.425625 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:32.425217 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:12:32.443882 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:32.443843 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-htxt8" event={"ID":"081e32e7-b1a0-49c3-87c6-1f690f0f09d7","Type":"ContainerStarted","Data":"9f0d9f8753e32bc7b94ff369579a105e201217e37ab3cd6127fa1d1fae207848"} Jul 31 14:12:32.476060 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:32.476024 2577 generic.go:356] "Generic (PLEG): container finished" podID="f2b4caf14cfc48bdf71cf245b1d50495" containerID="48a4755343502a070915268d0da14e0cecb534f19793b79652441daaf2442863" exitCode=0 Jul 31 14:12:32.476218 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:32.476123 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal" event={"ID":"f2b4caf14cfc48bdf71cf245b1d50495","Type":"ContainerDied","Data":"48a4755343502a070915268d0da14e0cecb534f19793b79652441daaf2442863"} Jul 31 14:12:32.516196 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:32.516121 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hclhf" event={"ID":"6a540518-e608-4021-a467-eaa910669373","Type":"ContainerStarted","Data":"22ff0a1eb8ae09f7fb57583adbee3f50df20049f385ac7b2856fc0bca48bbfb0"} Jul 31 14:12:32.528274 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:32.528159 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qltd5" event={"ID":"0fd2b5b1-f721-4a2e-8c19-1c220634d162","Type":"ContainerStarted","Data":"acb41868124c5c71c808a4797f794d97ddd1c366e7c04fa875fc65f180343650"} Jul 31 14:12:32.535871 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:32.535771 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" event={"ID":"37030756-473f-49b8-b47d-8e1e28df8150","Type":"ContainerStarted","Data":"78fa5cd91cf14bcd0f0e07ad0f99fba3ec03fca599d902c72469492dd1ba7cc8"} Jul 31 14:12:32.541918 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:32.541723 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-38.ec2.internal" event={"ID":"244611c7bbcc8364a26d08e8f81e9ee4","Type":"ContainerStarted","Data":"76c5b316ce52f29385130e5c9dc1f949e10f5324bb575b230a19269c7576ee33"} Jul 31 14:12:32.546139 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:32.546110 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9plr" event={"ID":"1a3824fc-d397-431c-8796-c494dc5d022b","Type":"ContainerStarted","Data":"f1fd4ae8851812c39cc8d35c6d4f8abff3396b4fc871b90b7d168368fdb2dc03"} Jul 31 14:12:32.561962 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:32.561906 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" event={"ID":"2d3c0ddd-40ab-42dd-ba02-186210158dcf","Type":"ContainerStarted","Data":"bacc5d075c963d71f26c5568a2cb8c3ea4acc27c0b4f2e090bb34c404bf090dc"} Jul 31 14:12:32.575554 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:32.573030 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wx97w" event={"ID":"c3047cea-d168-4307-a588-6c1f60e9a921","Type":"ContainerStarted","Data":"1f65f78627d0960e5458a49b65f9622b66f163b3e41c38fb1a75d539cd0722ca"} Jul 31 14:12:32.596113 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:32.596040 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" event={"ID":"ea37acc1-4e8e-4884-ae89-48c6aa635f03","Type":"ContainerStarted","Data":"fdb6acce35e5186ec2f87b31b7eb821bae8d70fcd1ba515259630c49cfd519c7"} Jul 31 14:12:32.601522 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:32.601472 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f2v62" event={"ID":"80dc0e3d-60fa-46e3-9e00-20eefb9986eb","Type":"ContainerStarted","Data":"52361fef0886eeb81ece79415991a34dac21e75024c0ea66d50e58c6a45a02d0"} Jul 31 14:12:33.425114 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:33.424430 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:33.425114 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:33.424557 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsrr4" podUID="72dad2d5-e7ca-4509-8619-e30eb609e348" Jul 31 14:12:33.425114 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:33.424965 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:33.425114 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:33.425064 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-229tg" podUID="0c82a3c9-4883-4803-9762-f008f3dfea21" Jul 31 14:12:33.622855 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:33.622789 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal" event={"ID":"f2b4caf14cfc48bdf71cf245b1d50495","Type":"ContainerStarted","Data":"ffff4c8596f60c88f022ffe0725c003187e09f6bfce76d3e373bb8eef398b75c"} Jul 31 14:12:33.636729 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:33.636678 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-38.ec2.internal" podStartSLOduration=4.636659677 podStartE2EDuration="4.636659677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:33.636592246 +0000 UTC m=+5.673415315" watchObservedRunningTime="2026-07-31 14:12:33.636659677 +0000 UTC m=+5.673482745" Jul 31 14:12:33.637207 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:33.637173 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-38.ec2.internal" podStartSLOduration=4.637162787 podStartE2EDuration="4.637162787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:32.557812559 +0000 UTC m=+4.594635632" watchObservedRunningTime="2026-07-31 14:12:33.637162787 +0000 UTC m=+5.673985855" Jul 31 14:12:33.933506 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:33.933470 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs\") pod \"network-metrics-daemon-skd9g\" (UID: \"78e66929-08a3-4aa1-8a96-740f8f8eb7c8\") " pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:33.933689 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:33.933527 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret\") pod \"global-pull-secret-syncer-lsrr4\" (UID: \"72dad2d5-e7ca-4509-8619-e30eb609e348\") " pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:33.933748 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:33.933703 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:33.933806 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:33.933763 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret podName:72dad2d5-e7ca-4509-8619-e30eb609e348 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:37.933744473 +0000 UTC m=+9.970567597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret") pod "global-pull-secret-syncer-lsrr4" (UID: "72dad2d5-e7ca-4509-8619-e30eb609e348") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:33.933953 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:33.933938 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:33.934013 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:33.933989 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs podName:78e66929-08a3-4aa1-8a96-740f8f8eb7c8 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:37.933972634 +0000 UTC m=+9.970795684 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs") pod "network-metrics-daemon-skd9g" (UID: "78e66929-08a3-4aa1-8a96-740f8f8eb7c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:34.034185 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:34.034148 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtps\" (UniqueName: \"kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps\") pod \"network-check-target-229tg\" (UID: \"0c82a3c9-4883-4803-9762-f008f3dfea21\") " pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:34.034448 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:34.034373 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:12:34.034448 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:34.034402 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:12:34.034448 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:34.034416 2577 projected.go:196] Error preparing data for projected volume kube-api-access-shtps for pod openshift-network-diagnostics/network-check-target-229tg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:34.034644 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:34.034478 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps podName:0c82a3c9-4883-4803-9762-f008f3dfea21 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:38.034460127 +0000 UTC m=+10.071283178 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-shtps" (UniqueName: "kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps") pod "network-check-target-229tg" (UID: "0c82a3c9-4883-4803-9762-f008f3dfea21") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:34.427302 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:34.427222 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:34.427724 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:34.427371 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:12:35.424762 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:35.424726 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:35.424965 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:35.424867 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsrr4" podUID="72dad2d5-e7ca-4509-8619-e30eb609e348" Jul 31 14:12:35.424965 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:35.424951 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:35.425081 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:35.425028 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-229tg" podUID="0c82a3c9-4883-4803-9762-f008f3dfea21" Jul 31 14:12:36.425409 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:36.425377 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:36.425811 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:36.425523 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:12:37.424553 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:37.424071 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:37.424553 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:37.424199 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-229tg" podUID="0c82a3c9-4883-4803-9762-f008f3dfea21" Jul 31 14:12:37.424553 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:37.424071 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:37.424553 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:37.424460 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsrr4" podUID="72dad2d5-e7ca-4509-8619-e30eb609e348" Jul 31 14:12:37.964878 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:37.964231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret\") pod \"global-pull-secret-syncer-lsrr4\" (UID: \"72dad2d5-e7ca-4509-8619-e30eb609e348\") " pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:37.964878 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:37.964315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs\") pod \"network-metrics-daemon-skd9g\" (UID: \"78e66929-08a3-4aa1-8a96-740f8f8eb7c8\") " pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:37.964878 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:37.964430 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:37.964878 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:37.964483 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs podName:78e66929-08a3-4aa1-8a96-740f8f8eb7c8 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:45.964469543 +0000 UTC m=+18.001292588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs") pod "network-metrics-daemon-skd9g" (UID: "78e66929-08a3-4aa1-8a96-740f8f8eb7c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:37.964878 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:37.964789 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:37.964878 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:37.964833 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret podName:72dad2d5-e7ca-4509-8619-e30eb609e348 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:45.964819644 +0000 UTC m=+18.001642693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret") pod "global-pull-secret-syncer-lsrr4" (UID: "72dad2d5-e7ca-4509-8619-e30eb609e348") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:38.064758 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:38.064684 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtps\" (UniqueName: \"kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps\") pod \"network-check-target-229tg\" (UID: \"0c82a3c9-4883-4803-9762-f008f3dfea21\") " pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:38.064965 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:38.064884 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:12:38.064965 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:38.064907 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:12:38.064965 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:38.064920 2577 projected.go:196] Error preparing data for projected volume kube-api-access-shtps for pod openshift-network-diagnostics/network-check-target-229tg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:38.065143 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:38.064979 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps podName:0c82a3c9-4883-4803-9762-f008f3dfea21 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:46.064960135 +0000 UTC m=+18.101783182 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-shtps" (UniqueName: "kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps") pod "network-check-target-229tg" (UID: "0c82a3c9-4883-4803-9762-f008f3dfea21") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:38.426124 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:38.425616 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:38.426124 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:38.425747 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:12:39.424262 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:39.424228 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:39.424716 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:39.424349 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-229tg" podUID="0c82a3c9-4883-4803-9762-f008f3dfea21" Jul 31 14:12:39.424716 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:39.424597 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:39.424809 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:39.424710 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsrr4" podUID="72dad2d5-e7ca-4509-8619-e30eb609e348" Jul 31 14:12:40.424957 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:40.424931 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:40.425358 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:40.425031 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:12:41.424212 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:41.424177 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:41.424402 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:41.424194 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:41.424402 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:41.424319 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsrr4" podUID="72dad2d5-e7ca-4509-8619-e30eb609e348" Jul 31 14:12:41.424402 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:41.424385 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-229tg" podUID="0c82a3c9-4883-4803-9762-f008f3dfea21" Jul 31 14:12:42.424373 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:42.424334 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:42.424868 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:42.424494 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:12:43.424315 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:43.424278 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:43.424315 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:43.424325 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:43.424860 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:43.424419 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsrr4" podUID="72dad2d5-e7ca-4509-8619-e30eb609e348" Jul 31 14:12:43.424860 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:43.424549 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-229tg" podUID="0c82a3c9-4883-4803-9762-f008f3dfea21" Jul 31 14:12:44.424803 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:44.424766 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:44.425245 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:44.424924 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:12:45.424185 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:45.424153 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:45.424399 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:45.424155 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:45.424399 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:45.424271 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-229tg" podUID="0c82a3c9-4883-4803-9762-f008f3dfea21" Jul 31 14:12:45.424399 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:45.424374 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsrr4" podUID="72dad2d5-e7ca-4509-8619-e30eb609e348" Jul 31 14:12:46.031620 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:46.031583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs\") pod \"network-metrics-daemon-skd9g\" (UID: \"78e66929-08a3-4aa1-8a96-740f8f8eb7c8\") " pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:46.031620 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:46.031626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret\") pod \"global-pull-secret-syncer-lsrr4\" (UID: \"72dad2d5-e7ca-4509-8619-e30eb609e348\") " pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:46.032081 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:46.031720 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:46.032081 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:46.031738 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:46.032081 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:46.031780 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret podName:72dad2d5-e7ca-4509-8619-e30eb609e348 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:02.031762636 +0000 UTC m=+34.068585682 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret") pod "global-pull-secret-syncer-lsrr4" (UID: "72dad2d5-e7ca-4509-8619-e30eb609e348") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:46.032081 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:46.031795 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs podName:78e66929-08a3-4aa1-8a96-740f8f8eb7c8 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:02.031788336 +0000 UTC m=+34.068611381 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs") pod "network-metrics-daemon-skd9g" (UID: "78e66929-08a3-4aa1-8a96-740f8f8eb7c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:46.132612 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:46.132577 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtps\" (UniqueName: \"kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps\") pod \"network-check-target-229tg\" (UID: \"0c82a3c9-4883-4803-9762-f008f3dfea21\") " pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:46.132787 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:46.132749 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:12:46.132787 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:46.132774 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:12:46.132787 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:46.132784 2577 projected.go:196] Error preparing data for projected volume kube-api-access-shtps for pod openshift-network-diagnostics/network-check-target-229tg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:46.132955 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:46.132833 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps podName:0c82a3c9-4883-4803-9762-f008f3dfea21 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:02.132818806 +0000 UTC m=+34.169641851 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-shtps" (UniqueName: "kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps") pod "network-check-target-229tg" (UID: "0c82a3c9-4883-4803-9762-f008f3dfea21") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:46.424825 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:46.424791 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:46.425002 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:46.424926 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:12:47.423960 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:47.423927 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:47.424377 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:47.424033 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsrr4" podUID="72dad2d5-e7ca-4509-8619-e30eb609e348" Jul 31 14:12:47.424377 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:47.424063 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:47.424377 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:47.424109 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-229tg" podUID="0c82a3c9-4883-4803-9762-f008f3dfea21" Jul 31 14:12:48.425483 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:48.425447 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:48.425962 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:48.425572 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:12:49.424600 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:49.424565 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:49.424791 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:49.424583 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:49.424791 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:49.424695 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsrr4" podUID="72dad2d5-e7ca-4509-8619-e30eb609e348" Jul 31 14:12:49.424791 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:49.424779 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-229tg" podUID="0c82a3c9-4883-4803-9762-f008f3dfea21" Jul 31 14:12:50.425088 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.424919 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:50.425692 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:50.425195 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:12:50.655787 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.655749 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hclhf" event={"ID":"6a540518-e608-4021-a467-eaa910669373","Type":"ContainerStarted","Data":"0a00b1759587bbb84d33e60a5d3241214ebf1cd50968734fae86618dd3916784"} Jul 31 14:12:50.657178 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.657150 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qltd5" event={"ID":"0fd2b5b1-f721-4a2e-8c19-1c220634d162","Type":"ContainerStarted","Data":"39c13506352fad79d57665ce5cf268c2483a2799d53a82f1e272d18be36aa5f1"} Jul 31 14:12:50.658609 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.658588 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" event={"ID":"37030756-473f-49b8-b47d-8e1e28df8150","Type":"ContainerStarted","Data":"f4977cdf176709ed532fbc0c5dbc1f51b136cbca833b861386eb1cafe567037a"} Jul 31 14:12:50.659926 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.659905 2577 generic.go:356] "Generic (PLEG): container finished" podID="1a3824fc-d397-431c-8796-c494dc5d022b" containerID="3c8b338141982d1dd88878795e7ec91306d0bbec78e970c9591749d333e9460f" exitCode=0 Jul 31 14:12:50.660027 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.659952 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9plr" event={"ID":"1a3824fc-d397-431c-8796-c494dc5d022b","Type":"ContainerDied","Data":"3c8b338141982d1dd88878795e7ec91306d0bbec78e970c9591749d333e9460f"} Jul 31 14:12:50.663112 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.663092 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" event={"ID":"2d3c0ddd-40ab-42dd-ba02-186210158dcf","Type":"ContainerStarted","Data":"fbfb635d01f2dbee0da343ec545a8bdd951d030b7c2e2c5abc629cd142e45b2c"} Jul 31 14:12:50.663185 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.663162 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" event={"ID":"2d3c0ddd-40ab-42dd-ba02-186210158dcf","Type":"ContainerStarted","Data":"6c5199bebee4901a9f546a28973529f7904b4c14cd3b5b724eea00dce0a8013f"} Jul 31 14:12:50.663239 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.663211 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" event={"ID":"2d3c0ddd-40ab-42dd-ba02-186210158dcf","Type":"ContainerStarted","Data":"bf37c7770a3f7c60bf24386961fdc729d2a2831e0048b2d4b6ad446c03358bd8"} Jul 31 14:12:50.663283 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.663241 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" event={"ID":"2d3c0ddd-40ab-42dd-ba02-186210158dcf","Type":"ContainerStarted","Data":"6af8998e0fb0e3d6308dee97b7c363690ad4106c29fad59377377a54b0651621"} Jul 31 14:12:50.663283 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.663274 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" event={"ID":"2d3c0ddd-40ab-42dd-ba02-186210158dcf","Type":"ContainerStarted","Data":"89560c8bae3bb61bdfaff0840ae139292dee626faa697f03aed38a711a4a71e9"} Jul 31 14:12:50.663382 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.663306 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" event={"ID":"2d3c0ddd-40ab-42dd-ba02-186210158dcf","Type":"ContainerStarted","Data":"3f919c5d431fc4296f3157faea904fb4a42439bd7abeebff60a0c81b62942a58"} Jul 31 14:12:50.664548 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.664529 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wx97w" event={"ID":"c3047cea-d168-4307-a588-6c1f60e9a921","Type":"ContainerStarted","Data":"4c17193d409372253593cc6fcdd1e58f36254b52c0c838384e4bb12529203d36"} Jul 31 14:12:50.665888 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.665868 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" event={"ID":"ea37acc1-4e8e-4884-ae89-48c6aa635f03","Type":"ContainerStarted","Data":"e370c90d37a33ad482532371952a403693e7015ebbd379d0e3966e3b3c8af22d"} Jul 31 14:12:50.667509 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.667490 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f2v62" event={"ID":"80dc0e3d-60fa-46e3-9e00-20eefb9986eb","Type":"ContainerStarted","Data":"a5a63c77d439eef256f91d47d8c05c46e225368f23fa67d0a5007753e298a236"} Jul 31 14:12:50.667973 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.667941 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hclhf" podStartSLOduration=12.21407981 podStartE2EDuration="22.667930462s" totalImagesPullingTime="10.453850652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:50.667778682 +0000 UTC m=+22.704601748" watchObservedRunningTime="2026-07-31 14:12:50.667930462 +0000 UTC m=+22.704753528" Jul 31 14:12:50.680580 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.680545 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-f2v62" podStartSLOduration=4.379407095 podStartE2EDuration="22.680533455s" totalImagesPullingTime="18.30112636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:50.680341009 +0000 UTC m=+22.717164067" watchObservedRunningTime="2026-07-31 14:12:50.680533455 +0000 UTC m=+22.717356522" Jul 31 14:12:50.696676 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.696638 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wzqr6" podStartSLOduration=4.395236816 podStartE2EDuration="22.696630862s" totalImagesPullingTime="18.301394046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:50.696237613 +0000 UTC m=+22.733060679" watchObservedRunningTime="2026-07-31 14:12:50.696630862 +0000 UTC m=+22.733453928" Jul 31 14:12:50.728752 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.728698 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qltd5" podStartSLOduration=4.438146741 podStartE2EDuration="22.728683623s" totalImagesPullingTime="18.290536882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:50.728307615 +0000 UTC m=+22.765130692" watchObservedRunningTime="2026-07-31 14:12:50.728683623 +0000 UTC m=+22.765506691" Jul 31 14:12:50.743509 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:50.743471 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wx97w" podStartSLOduration=4.400642597 podStartE2EDuration="22.743458961s" totalImagesPullingTime="18.342816364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:50.743054644 +0000 UTC m=+22.779877836" watchObservedRunningTime="2026-07-31 14:12:50.743458961 +0000 UTC m=+22.780282027" Jul 31 14:12:51.233556 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:51.233319 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hclhf" Jul 31 14:12:51.234235 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:51.234215 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hclhf" Jul 31 14:12:51.326771 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:51.326749 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 31 14:12:51.361992 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:51.361905 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T14:12:51.32676685Z","UUID":"16068080-9c6b-474b-a57f-b1474047d9db","Handler":null,"Name":"","Endpoint":""} Jul 31 14:12:51.363247 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:51.363230 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 31 14:12:51.363354 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:51.363266 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 31 14:12:51.424115 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:51.424047 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:51.424329 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:51.424047 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:51.424329 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:51.424142 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsrr4" podUID="72dad2d5-e7ca-4509-8619-e30eb609e348" Jul 31 14:12:51.424329 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:51.424236 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-229tg" podUID="0c82a3c9-4883-4803-9762-f008f3dfea21" Jul 31 14:12:51.671250 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:51.671215 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-htxt8" event={"ID":"081e32e7-b1a0-49c3-87c6-1f690f0f09d7","Type":"ContainerStarted","Data":"7410a26c369ea8dd51f9712f4688ff42639cc085a150a4e82c034a4f12034b26"} Jul 31 14:12:51.672877 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:51.672855 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" event={"ID":"37030756-473f-49b8-b47d-8e1e28df8150","Type":"ContainerStarted","Data":"b9a516cae83f7536df4d44d720f92554a7af413c2f74f67536d48ebcb6038a2b"} Jul 31 14:12:51.685900 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:51.685816 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-htxt8" podStartSLOduration=5.383463775 podStartE2EDuration="23.685805461s" totalImagesPullingTime="18.302341686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:51.68550819 +0000 UTC m=+23.722331257" watchObservedRunningTime="2026-07-31 14:12:51.685805461 +0000 UTC m=+23.722628577" Jul 31 14:12:52.424457 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:52.424427 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:52.424634 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:52.424562 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:12:52.681530 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:52.681458 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" event={"ID":"2d3c0ddd-40ab-42dd-ba02-186210158dcf","Type":"ContainerStarted","Data":"bb6555bd503fbd6e136aafca787ed3c569a1c2a2f85e4392df842bd3ef99fdd2"} Jul 31 14:12:52.683438 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:52.683411 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" event={"ID":"37030756-473f-49b8-b47d-8e1e28df8150","Type":"ContainerStarted","Data":"16589d7347ee664f19d3a2d783ff150a9d229709a6c514c3c9800f8883699fdf"} Jul 31 14:12:52.683734 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:52.683715 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 31 14:12:53.424244 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:53.424217 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:53.424244 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:53.424233 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:53.424467 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:53.424317 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-229tg" podUID="0c82a3c9-4883-4803-9762-f008f3dfea21" Jul 31 14:12:53.424467 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:53.424443 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsrr4" podUID="72dad2d5-e7ca-4509-8619-e30eb609e348" Jul 31 14:12:54.425018 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:54.424835 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:54.425374 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:54.425107 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:12:54.504210 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:54.504175 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hclhf" Jul 31 14:12:54.504388 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:54.504314 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 31 14:12:54.504823 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:54.504801 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hclhf" Jul 31 14:12:54.521246 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:54.521193 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmw4w" podStartSLOduration=5.908208798 podStartE2EDuration="26.521178351s" totalImagesPullingTime="20.612969553s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:52.698561808 +0000 UTC m=+24.735384876" watchObservedRunningTime="2026-07-31 14:12:54.521178351 +0000 UTC m=+26.558001417" Jul 31 14:12:55.424785 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:55.424731 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:55.424982 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:55.424868 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-229tg" podUID="0c82a3c9-4883-4803-9762-f008f3dfea21" Jul 31 14:12:55.424982 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:55.424908 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:55.425364 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:55.425006 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsrr4" podUID="72dad2d5-e7ca-4509-8619-e30eb609e348" Jul 31 14:12:56.424571 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:56.424312 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:56.424734 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:56.424708 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:12:56.694639 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:56.694604 2577 generic.go:356] "Generic (PLEG): container finished" podID="1a3824fc-d397-431c-8796-c494dc5d022b" containerID="69c8474022f200e14bace6ddce0ee3bbd02931b5855f8f90cf607abec3c1b140" exitCode=0 Jul 31 14:12:56.695512 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:56.694683 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9plr" event={"ID":"1a3824fc-d397-431c-8796-c494dc5d022b","Type":"ContainerDied","Data":"69c8474022f200e14bace6ddce0ee3bbd02931b5855f8f90cf607abec3c1b140"} Jul 31 14:12:56.698438 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:56.698415 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" event={"ID":"2d3c0ddd-40ab-42dd-ba02-186210158dcf","Type":"ContainerStarted","Data":"a31429a586acb968b42fa081b0bff4aea194094c9aaddd101f70fe3b0d5984fb"} Jul 31 14:12:56.698725 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:56.698707 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:56.698816 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:56.698770 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:56.713403 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:56.713376 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:56.713512 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:56.713498 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:56.736521 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:56.736479 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" podStartSLOduration=10.380461526 podStartE2EDuration="28.736466366s" totalImagesPullingTime="18.35600484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:56.735633103 +0000 UTC m=+28.772456171" watchObservedRunningTime="2026-07-31 14:12:56.736466366 +0000 UTC m=+28.773289434" Jul 31 14:12:57.424442 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:57.424411 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:57.424677 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:57.424411 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:57.424677 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:57.424533 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsrr4" podUID="72dad2d5-e7ca-4509-8619-e30eb609e348" Jul 31 14:12:57.424677 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:57.424649 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-229tg" podUID="0c82a3c9-4883-4803-9762-f008f3dfea21" Jul 31 14:12:57.700159 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:57.700100 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 31 14:12:58.080507 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:58.080433 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lsrr4"] Jul 31 14:12:58.080663 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:58.080521 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:58.080663 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:58.080600 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsrr4" podUID="72dad2d5-e7ca-4509-8619-e30eb609e348" Jul 31 14:12:58.081114 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:58.081090 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-skd9g"] Jul 31 14:12:58.081228 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:58.081194 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:58.081318 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:58.081299 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:12:58.082668 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:58.082648 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-229tg"] Jul 31 14:12:58.082742 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:58.082715 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:58.082795 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:58.082779 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-229tg" podUID="0c82a3c9-4883-4803-9762-f008f3dfea21" Jul 31 14:12:58.427017 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:58.426988 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:12:58.704961 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:58.704872 2577 generic.go:356] "Generic (PLEG): container finished" podID="1a3824fc-d397-431c-8796-c494dc5d022b" containerID="1abf4a8439e750e0b6d51fe36a84e6452a8eaf60586b2e5c14a8899c0751d40e" exitCode=0 Jul 31 14:12:58.704961 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:58.704944 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9plr" event={"ID":"1a3824fc-d397-431c-8796-c494dc5d022b","Type":"ContainerDied","Data":"1abf4a8439e750e0b6d51fe36a84e6452a8eaf60586b2e5c14a8899c0751d40e"} Jul 31 14:12:59.424586 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:59.424416 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:12:59.424755 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:59.424467 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:12:59.424755 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:59.424646 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsrr4" podUID="72dad2d5-e7ca-4509-8619-e30eb609e348" Jul 31 14:12:59.424755 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:59.424740 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-229tg" podUID="0c82a3c9-4883-4803-9762-f008f3dfea21" Jul 31 14:12:59.424958 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:12:59.424475 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:12:59.424958 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:12:59.424912 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:13:00.711411 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:00.711381 2577 generic.go:356] "Generic (PLEG): container finished" podID="1a3824fc-d397-431c-8796-c494dc5d022b" containerID="23e3b463984c5eb71fa6b94cd6cde3423b4d3147af8050bf88b0997257ed035f" exitCode=0 Jul 31 14:13:00.711786 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:00.711452 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9plr" event={"ID":"1a3824fc-d397-431c-8796-c494dc5d022b","Type":"ContainerDied","Data":"23e3b463984c5eb71fa6b94cd6cde3423b4d3147af8050bf88b0997257ed035f"} Jul 31 14:13:01.424693 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:01.424606 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:13:01.424693 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:01.424654 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:13:01.424933 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:01.424745 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsrr4" podUID="72dad2d5-e7ca-4509-8619-e30eb609e348" Jul 31 14:13:01.424933 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:01.424796 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:13:01.424933 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:01.424903 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-229tg" podUID="0c82a3c9-4883-4803-9762-f008f3dfea21" Jul 31 14:13:01.425092 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:01.425013 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:13:02.052752 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.052716 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs\") pod \"network-metrics-daemon-skd9g\" (UID: \"78e66929-08a3-4aa1-8a96-740f8f8eb7c8\") " pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:13:02.053336 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.052765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret\") pod \"global-pull-secret-syncer-lsrr4\" (UID: \"72dad2d5-e7ca-4509-8619-e30eb609e348\") " pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:13:02.053336 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:02.052831 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:13:02.053336 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:02.052888 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:13:02.053336 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:02.052920 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs podName:78e66929-08a3-4aa1-8a96-740f8f8eb7c8 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:34.052897262 +0000 UTC m=+66.089720312 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs") pod "network-metrics-daemon-skd9g" (UID: "78e66929-08a3-4aa1-8a96-740f8f8eb7c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:13:02.053336 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:02.052940 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret podName:72dad2d5-e7ca-4509-8619-e30eb609e348 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:34.052930428 +0000 UTC m=+66.089753478 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret") pod "global-pull-secret-syncer-lsrr4" (UID: "72dad2d5-e7ca-4509-8619-e30eb609e348") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:13:02.153250 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.153216 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtps\" (UniqueName: \"kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps\") pod \"network-check-target-229tg\" (UID: \"0c82a3c9-4883-4803-9762-f008f3dfea21\") " pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:13:02.153423 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:02.153343 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:13:02.153423 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:02.153358 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:13:02.153423 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:02.153367 2577 projected.go:196] Error preparing data for projected volume kube-api-access-shtps for pod openshift-network-diagnostics/network-check-target-229tg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:13:02.153423 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:02.153419 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps podName:0c82a3c9-4883-4803-9762-f008f3dfea21 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:34.15340121 +0000 UTC m=+66.190224256 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-shtps" (UniqueName: "kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps") pod "network-check-target-229tg" (UID: "0c82a3c9-4883-4803-9762-f008f3dfea21") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:13:02.759329 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.759300 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-38.ec2.internal" event="NodeReady" Jul 31 14:13:02.759504 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.759450 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 31 14:13:02.801389 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.801351 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6lm6j"] Jul 31 14:13:02.804046 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.804023 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6lm6j" Jul 31 14:13:02.805982 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.805956 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8l2tx"] Jul 31 14:13:02.806154 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.806114 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 31 14:13:02.806311 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.806289 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 31 14:13:02.806371 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.806322 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p9vdf\"" Jul 31 14:13:02.807945 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.807926 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8l2tx" Jul 31 14:13:02.809631 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.809613 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p49zl\"" Jul 31 14:13:02.809753 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.809642 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 31 14:13:02.809962 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.809947 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 31 14:13:02.810038 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.809947 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 31 14:13:02.813425 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.813171 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6lm6j"] Jul 31 14:13:02.816020 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.815999 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8l2tx"] Jul 31 14:13:02.959956 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.959925 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5v4p\" (UniqueName: \"kubernetes.io/projected/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-kube-api-access-j5v4p\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:13:02.960117 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.959962 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43e35457-db4f-45d4-a5ac-dd16537435d6-cert\") pod \"ingress-canary-8l2tx\" (UID: \"43e35457-db4f-45d4-a5ac-dd16537435d6\") " pod="openshift-ingress-canary/ingress-canary-8l2tx" Jul 31 14:13:02.960117 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.959988 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-config-volume\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:13:02.960117 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.960065 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsxd6\" (UniqueName: \"kubernetes.io/projected/43e35457-db4f-45d4-a5ac-dd16537435d6-kube-api-access-fsxd6\") pod \"ingress-canary-8l2tx\" (UID: \"43e35457-db4f-45d4-a5ac-dd16537435d6\") " pod="openshift-ingress-canary/ingress-canary-8l2tx" Jul 31 14:13:02.960117 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.960113 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-tmp-dir\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:13:02.960271 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:02.960159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:13:03.061278 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.061133 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5v4p\" (UniqueName: \"kubernetes.io/projected/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-kube-api-access-j5v4p\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:13:03.061278 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.061175 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43e35457-db4f-45d4-a5ac-dd16537435d6-cert\") pod \"ingress-canary-8l2tx\" (UID: \"43e35457-db4f-45d4-a5ac-dd16537435d6\") " pod="openshift-ingress-canary/ingress-canary-8l2tx" Jul 31 14:13:03.061278 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.061201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-config-volume\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:13:03.061278 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.061254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsxd6\" (UniqueName: \"kubernetes.io/projected/43e35457-db4f-45d4-a5ac-dd16537435d6-kube-api-access-fsxd6\") pod \"ingress-canary-8l2tx\" (UID: \"43e35457-db4f-45d4-a5ac-dd16537435d6\") " pod="openshift-ingress-canary/ingress-canary-8l2tx" Jul 31 14:13:03.061984 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.061288 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-tmp-dir\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:13:03.061984 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.061330 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:13:03.061984 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:03.061477 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:13:03.061984 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:03.061571 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls podName:7655bcd6-8ab3-4f31-bd58-7ebfef6f840b nodeName:}" failed. No retries permitted until 2026-07-31 14:13:03.561553359 +0000 UTC m=+35.598376420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls") pod "dns-default-6lm6j" (UID: "7655bcd6-8ab3-4f31-bd58-7ebfef6f840b") : secret "dns-default-metrics-tls" not found Jul 31 14:13:03.061984 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.061862 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-config-volume\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:13:03.062451 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.062430 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-tmp-dir\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:13:03.066085 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.066059 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43e35457-db4f-45d4-a5ac-dd16537435d6-cert\") pod \"ingress-canary-8l2tx\" (UID: \"43e35457-db4f-45d4-a5ac-dd16537435d6\") " pod="openshift-ingress-canary/ingress-canary-8l2tx" Jul 31 14:13:03.071502 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.071429 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsxd6\" (UniqueName: \"kubernetes.io/projected/43e35457-db4f-45d4-a5ac-dd16537435d6-kube-api-access-fsxd6\") pod \"ingress-canary-8l2tx\" (UID: \"43e35457-db4f-45d4-a5ac-dd16537435d6\") " pod="openshift-ingress-canary/ingress-canary-8l2tx" Jul 31 14:13:03.071502 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.071480 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5v4p\" (UniqueName: \"kubernetes.io/projected/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-kube-api-access-j5v4p\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:13:03.124735 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.124691 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8l2tx" Jul 31 14:13:03.424859 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.424800 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:13:03.424859 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.424856 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:13:03.425088 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.424932 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:13:03.430374 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.430348 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 31 14:13:03.430773 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.430752 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lwbh\"" Jul 31 14:13:03.431066 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.431018 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 31 14:13:03.431171 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.431070 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 31 14:13:03.431221 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.431166 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 31 14:13:03.431221 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.431197 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rdxfb\"" Jul 31 14:13:03.565502 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:03.565467 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:13:03.565685 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:03.565620 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:13:03.565685 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:03.565682 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls podName:7655bcd6-8ab3-4f31-bd58-7ebfef6f840b nodeName:}" failed. No retries permitted until 2026-07-31 14:13:04.565667424 +0000 UTC m=+36.602490470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls") pod "dns-default-6lm6j" (UID: "7655bcd6-8ab3-4f31-bd58-7ebfef6f840b") : secret "dns-default-metrics-tls" not found Jul 31 14:13:04.573481 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:04.573440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:13:04.574151 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:04.573631 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:13:04.574151 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:04.573715 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls podName:7655bcd6-8ab3-4f31-bd58-7ebfef6f840b nodeName:}" failed. No retries permitted until 2026-07-31 14:13:06.573693709 +0000 UTC m=+38.610516758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls") pod "dns-default-6lm6j" (UID: "7655bcd6-8ab3-4f31-bd58-7ebfef6f840b") : secret "dns-default-metrics-tls" not found Jul 31 14:13:04.737781 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:04.737575 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8l2tx"] Jul 31 14:13:04.741267 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:13:04.741241 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e35457_db4f_45d4_a5ac_dd16537435d6.slice/crio-feb7752451f42242bad95d1f24efcdba9ac7a2a55402ca6e78d65b5c7119ae27 WatchSource:0}: Error finding container feb7752451f42242bad95d1f24efcdba9ac7a2a55402ca6e78d65b5c7119ae27: Status 404 returned error can't find the container with id feb7752451f42242bad95d1f24efcdba9ac7a2a55402ca6e78d65b5c7119ae27 Jul 31 14:13:05.726009 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:05.725975 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8l2tx" event={"ID":"43e35457-db4f-45d4-a5ac-dd16537435d6","Type":"ContainerStarted","Data":"feb7752451f42242bad95d1f24efcdba9ac7a2a55402ca6e78d65b5c7119ae27"} Jul 31 14:13:05.729148 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:05.729117 2577 generic.go:356] "Generic (PLEG): container finished" podID="1a3824fc-d397-431c-8796-c494dc5d022b" containerID="e05533c60bbef8fc52dd2efd102f520537fc2796b7c807ef5332863de7c93ed8" exitCode=0 Jul 31 14:13:05.729278 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:05.729162 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9plr" event={"ID":"1a3824fc-d397-431c-8796-c494dc5d022b","Type":"ContainerDied","Data":"e05533c60bbef8fc52dd2efd102f520537fc2796b7c807ef5332863de7c93ed8"} Jul 31 14:13:06.589999 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:06.589937 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:13:06.590141 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:06.590093 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:13:06.590190 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:06.590157 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls podName:7655bcd6-8ab3-4f31-bd58-7ebfef6f840b nodeName:}" failed. No retries permitted until 2026-07-31 14:13:10.590140826 +0000 UTC m=+42.626963871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls") pod "dns-default-6lm6j" (UID: "7655bcd6-8ab3-4f31-bd58-7ebfef6f840b") : secret "dns-default-metrics-tls" not found Jul 31 14:13:06.735610 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:06.735571 2577 generic.go:356] "Generic (PLEG): container finished" podID="1a3824fc-d397-431c-8796-c494dc5d022b" containerID="676aa27510456eae1c4f2b7eb0d9a1d7ed1e6e2fd7083d544397dd7d00814ef8" exitCode=0 Jul 31 14:13:06.736076 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:06.735657 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9plr" event={"ID":"1a3824fc-d397-431c-8796-c494dc5d022b","Type":"ContainerDied","Data":"676aa27510456eae1c4f2b7eb0d9a1d7ed1e6e2fd7083d544397dd7d00814ef8"} Jul 31 14:13:07.741211 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:07.741132 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9plr" event={"ID":"1a3824fc-d397-431c-8796-c494dc5d022b","Type":"ContainerStarted","Data":"a72a7a39a4c8a26ff03acbeac27eae5b6295599b159969db001d22f8fc65dd33"} Jul 31 14:13:07.762081 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:07.761814 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g9plr" podStartSLOduration=9.981002382 podStartE2EDuration="39.761796067s" totalImagesPullingTime="29.780793685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:07.760163247 +0000 UTC m=+39.796986313" watchObservedRunningTime="2026-07-31 14:13:07.761796067 +0000 UTC m=+39.798619134" Jul 31 14:13:08.744242 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:08.744205 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8l2tx" event={"ID":"43e35457-db4f-45d4-a5ac-dd16537435d6","Type":"ContainerStarted","Data":"2acf8ce58049f937e689c3786ea1d77aa4cbf53c5801b3bcaa7697fb90afee6b"} Jul 31 14:13:08.758848 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:08.758806 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8l2tx" podStartSLOduration=3.7462905810000002 podStartE2EDuration="6.758790832s" totalImagesPullingTime="3.012500251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:13:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:08.758242091 +0000 UTC m=+40.795065158" watchObservedRunningTime="2026-07-31 14:13:08.758790832 +0000 UTC m=+40.795613897" Jul 31 14:13:10.095288 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.095251 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk"] Jul 31 14:13:10.130856 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.130811 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z"] Jul 31 14:13:10.131024 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.130943 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk" Jul 31 14:13:10.133071 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.133041 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 31 14:13:10.133276 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.133257 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-wgt54\"" Jul 31 14:13:10.133654 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.133636 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 31 14:13:10.133762 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.133724 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 31 14:13:10.133819 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.133757 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 31 14:13:10.146208 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.146185 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk"] Jul 31 14:13:10.146323 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.146243 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z"] Jul 31 14:13:10.146389 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.146339 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.148178 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.148151 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 31 14:13:10.148285 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.148202 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 31 14:13:10.148285 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.148264 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 31 14:13:10.148424 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.148412 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 31 14:13:10.214515 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.214478 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-684c8745c8-rcz6z\" (UID: \"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.214710 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.214533 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44wdf\" (UniqueName: \"kubernetes.io/projected/5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964-kube-api-access-44wdf\") pod \"cluster-proxy-proxy-agent-684c8745c8-rcz6z\" (UID: \"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.214710 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.214615 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964-ca\") pod \"cluster-proxy-proxy-agent-684c8745c8-rcz6z\" (UID: \"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.214710 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.214668 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964-additional-service-ca\") pod \"cluster-proxy-proxy-agent-684c8745c8-rcz6z\" (UID: \"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.214710 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.214700 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnm9d\" (UniqueName: \"kubernetes.io/projected/1d929dcc-18d9-4928-a93e-bd8fdb77100e-kube-api-access-jnm9d\") pod \"managed-serviceaccount-addon-agent-6fc587fd96-cqpjk\" (UID: \"1d929dcc-18d9-4928-a93e-bd8fdb77100e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk" Jul 31 14:13:10.214929 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.214748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964-hub\") pod \"cluster-proxy-proxy-agent-684c8745c8-rcz6z\" (UID: \"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.214929 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.214776 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-684c8745c8-rcz6z\" (UID: \"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.214929 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.214805 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1d929dcc-18d9-4928-a93e-bd8fdb77100e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fc587fd96-cqpjk\" (UID: \"1d929dcc-18d9-4928-a93e-bd8fdb77100e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk" Jul 31 14:13:10.315926 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.315891 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-684c8745c8-rcz6z\" (UID: \"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.316118 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.315949 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44wdf\" (UniqueName: \"kubernetes.io/projected/5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964-kube-api-access-44wdf\") pod \"cluster-proxy-proxy-agent-684c8745c8-rcz6z\" (UID: \"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.316118 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.315989 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964-ca\") pod \"cluster-proxy-proxy-agent-684c8745c8-rcz6z\" (UID: \"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.316118 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.316011 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964-additional-service-ca\") pod \"cluster-proxy-proxy-agent-684c8745c8-rcz6z\" (UID: \"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.316118 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.316036 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnm9d\" (UniqueName: \"kubernetes.io/projected/1d929dcc-18d9-4928-a93e-bd8fdb77100e-kube-api-access-jnm9d\") pod \"managed-serviceaccount-addon-agent-6fc587fd96-cqpjk\" (UID: \"1d929dcc-18d9-4928-a93e-bd8fdb77100e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk" Jul 31 14:13:10.316118 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.316075 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964-hub\") pod \"cluster-proxy-proxy-agent-684c8745c8-rcz6z\" (UID: \"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.316118 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.316096 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-684c8745c8-rcz6z\" (UID: \"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.316419 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.316123 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1d929dcc-18d9-4928-a93e-bd8fdb77100e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fc587fd96-cqpjk\" (UID: \"1d929dcc-18d9-4928-a93e-bd8fdb77100e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk" Jul 31 14:13:10.317089 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.317042 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964-additional-service-ca\") pod \"cluster-proxy-proxy-agent-684c8745c8-rcz6z\" (UID: \"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.320287 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.320262 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-684c8745c8-rcz6z\" (UID: \"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.320392 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.320268 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964-ca\") pod \"cluster-proxy-proxy-agent-684c8745c8-rcz6z\" (UID: \"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.320392 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.320350 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-684c8745c8-rcz6z\" (UID: \"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.320457 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.320386 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964-hub\") pod \"cluster-proxy-proxy-agent-684c8745c8-rcz6z\" (UID: \"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.320457 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.320441 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1d929dcc-18d9-4928-a93e-bd8fdb77100e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fc587fd96-cqpjk\" (UID: \"1d929dcc-18d9-4928-a93e-bd8fdb77100e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk" Jul 31 14:13:10.323522 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.323501 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnm9d\" (UniqueName: \"kubernetes.io/projected/1d929dcc-18d9-4928-a93e-bd8fdb77100e-kube-api-access-jnm9d\") pod \"managed-serviceaccount-addon-agent-6fc587fd96-cqpjk\" (UID: \"1d929dcc-18d9-4928-a93e-bd8fdb77100e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk" Jul 31 14:13:10.323619 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.323544 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44wdf\" (UniqueName: \"kubernetes.io/projected/5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964-kube-api-access-44wdf\") pod \"cluster-proxy-proxy-agent-684c8745c8-rcz6z\" (UID: \"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.452337 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.452307 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk" Jul 31 14:13:10.458456 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.458433 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:13:10.618428 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.618397 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:13:10.618588 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:10.618567 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:13:10.618646 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:10.618635 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls podName:7655bcd6-8ab3-4f31-bd58-7ebfef6f840b nodeName:}" failed. No retries permitted until 2026-07-31 14:13:18.61861518 +0000 UTC m=+50.655438237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls") pod "dns-default-6lm6j" (UID: "7655bcd6-8ab3-4f31-bd58-7ebfef6f840b") : secret "dns-default-metrics-tls" not found Jul 31 14:13:10.621331 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.621305 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk"] Jul 31 14:13:10.624692 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:13:10.624663 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d929dcc_18d9_4928_a93e_bd8fdb77100e.slice/crio-7a86a26317d5894c8e81d8e1f266cb2c2ac3bd8ce06504eac661a5c0cbb1253f WatchSource:0}: Error finding container 7a86a26317d5894c8e81d8e1f266cb2c2ac3bd8ce06504eac661a5c0cbb1253f: Status 404 returned error can't find the container with id 7a86a26317d5894c8e81d8e1f266cb2c2ac3bd8ce06504eac661a5c0cbb1253f Jul 31 14:13:10.640158 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.640135 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z"] Jul 31 14:13:10.645419 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:13:10.645391 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5feafd4d_47c7_4c7b_b6ec_eafbb6f6d964.slice/crio-664de013fabb80dcf3d24a88eb5e5774931dbd3407e8c8305cea5b0dc6c34509 WatchSource:0}: Error finding container 664de013fabb80dcf3d24a88eb5e5774931dbd3407e8c8305cea5b0dc6c34509: Status 404 returned error can't find the container with id 664de013fabb80dcf3d24a88eb5e5774931dbd3407e8c8305cea5b0dc6c34509 Jul 31 14:13:10.750412 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.750321 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" event={"ID":"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964","Type":"ContainerStarted","Data":"664de013fabb80dcf3d24a88eb5e5774931dbd3407e8c8305cea5b0dc6c34509"} Jul 31 14:13:10.751382 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:10.751358 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk" event={"ID":"1d929dcc-18d9-4928-a93e-bd8fdb77100e","Type":"ContainerStarted","Data":"7a86a26317d5894c8e81d8e1f266cb2c2ac3bd8ce06504eac661a5c0cbb1253f"} Jul 31 14:13:16.768737 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:16.768646 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk" event={"ID":"1d929dcc-18d9-4928-a93e-bd8fdb77100e","Type":"ContainerStarted","Data":"4905c282265edf760ef251becf7ef5af6ec29a07489ae5f4b03861c34d569068"} Jul 31 14:13:16.770735 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:16.770706 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" event={"ID":"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964","Type":"ContainerStarted","Data":"8d66ba99efde0ace7aa78277008ae36f9f9fca8ddabef7133035d142d91979b1"} Jul 31 14:13:16.770875 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:16.770746 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" event={"ID":"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964","Type":"ContainerStarted","Data":"a970f117b80ae481cbc541b29ceac625751e06b475e08aaee45ff1dc35718b76"} Jul 31 14:13:16.770875 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:16.770765 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" event={"ID":"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964","Type":"ContainerStarted","Data":"a7e6cec3c377e631800697cf280eea395eb0948eff8de85c8619b3925581ccaa"} Jul 31 14:13:16.783568 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:16.783518 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk" podStartSLOduration=1.160199532 podStartE2EDuration="6.783502768s" totalImagesPullingTime="5.623303236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:16.78309764 +0000 UTC m=+48.819920707" watchObservedRunningTime="2026-07-31 14:13:16.783502768 +0000 UTC m=+48.820325835" Jul 31 14:13:16.800352 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:16.800311 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" podStartSLOduration=1.181370069 podStartE2EDuration="6.800293079s" totalImagesPullingTime="5.61892301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:16.79974833 +0000 UTC m=+48.836571439" watchObservedRunningTime="2026-07-31 14:13:16.800293079 +0000 UTC m=+48.837116147" Jul 31 14:13:18.682900 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:18.682832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:13:18.683302 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:18.682988 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:13:18.683302 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:18.683058 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls podName:7655bcd6-8ab3-4f31-bd58-7ebfef6f840b nodeName:}" failed. No retries permitted until 2026-07-31 14:13:34.683042212 +0000 UTC m=+66.719865257 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls") pod "dns-default-6lm6j" (UID: "7655bcd6-8ab3-4f31-bd58-7ebfef6f840b") : secret "dns-default-metrics-tls" not found Jul 31 14:13:29.719207 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:29.719173 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sml2" Jul 31 14:13:34.093810 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:34.093753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs\") pod \"network-metrics-daemon-skd9g\" (UID: \"78e66929-08a3-4aa1-8a96-740f8f8eb7c8\") " pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:13:34.093810 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:34.093816 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret\") pod \"global-pull-secret-syncer-lsrr4\" (UID: \"72dad2d5-e7ca-4509-8619-e30eb609e348\") " pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:13:34.096367 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:34.096338 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 31 14:13:34.096475 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:34.096381 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 31 14:13:34.104849 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:34.104821 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 31 14:13:34.104917 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:34.104892 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs podName:78e66929-08a3-4aa1-8a96-740f8f8eb7c8 nodeName:}" failed. No retries permitted until 2026-07-31 14:14:38.104874997 +0000 UTC m=+130.141698043 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs") pod "network-metrics-daemon-skd9g" (UID: "78e66929-08a3-4aa1-8a96-740f8f8eb7c8") : secret "metrics-daemon-secret" not found Jul 31 14:13:34.106997 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:34.106975 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72dad2d5-e7ca-4509-8619-e30eb609e348-original-pull-secret\") pod \"global-pull-secret-syncer-lsrr4\" (UID: \"72dad2d5-e7ca-4509-8619-e30eb609e348\") " pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:13:34.194902 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:34.194871 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtps\" (UniqueName: \"kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps\") pod \"network-check-target-229tg\" (UID: \"0c82a3c9-4883-4803-9762-f008f3dfea21\") " pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:13:34.197254 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:34.197229 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 31 14:13:34.206571 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:34.206547 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 31 14:13:34.218243 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:34.218213 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shtps\" (UniqueName: \"kubernetes.io/projected/0c82a3c9-4883-4803-9762-f008f3dfea21-kube-api-access-shtps\") pod \"network-check-target-229tg\" (UID: \"0c82a3c9-4883-4803-9762-f008f3dfea21\") " pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:13:34.339865 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:34.339798 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsrr4" Jul 31 14:13:34.355172 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:34.355147 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rdxfb\"" Jul 31 14:13:34.363720 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:34.363699 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:13:34.484134 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:34.484101 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lsrr4"] Jul 31 14:13:34.487588 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:13:34.487551 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72dad2d5_e7ca_4509_8619_e30eb609e348.slice/crio-d77abd464d81f37c3c7ddb48a0ab88bdfce06f2e136c28f67462266448438bad WatchSource:0}: Error finding container d77abd464d81f37c3c7ddb48a0ab88bdfce06f2e136c28f67462266448438bad: Status 404 returned error can't find the container with id d77abd464d81f37c3c7ddb48a0ab88bdfce06f2e136c28f67462266448438bad Jul 31 14:13:34.495240 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:34.495213 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-229tg"] Jul 31 14:13:34.499370 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:13:34.499348 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c82a3c9_4883_4803_9762_f008f3dfea21.slice/crio-c06b622a263aeef146a70789229d354dfc5bd81fb158b7414668aab99948af3d WatchSource:0}: Error finding container c06b622a263aeef146a70789229d354dfc5bd81fb158b7414668aab99948af3d: Status 404 returned error can't find the container with id c06b622a263aeef146a70789229d354dfc5bd81fb158b7414668aab99948af3d Jul 31 14:13:34.699192 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:34.699151 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:13:34.699360 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:34.699276 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:13:34.699360 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:13:34.699328 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls podName:7655bcd6-8ab3-4f31-bd58-7ebfef6f840b nodeName:}" failed. No retries permitted until 2026-07-31 14:14:06.699315411 +0000 UTC m=+98.736138456 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls") pod "dns-default-6lm6j" (UID: "7655bcd6-8ab3-4f31-bd58-7ebfef6f840b") : secret "dns-default-metrics-tls" not found Jul 31 14:13:34.814779 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:34.814744 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lsrr4" event={"ID":"72dad2d5-e7ca-4509-8619-e30eb609e348","Type":"ContainerStarted","Data":"d77abd464d81f37c3c7ddb48a0ab88bdfce06f2e136c28f67462266448438bad"} Jul 31 14:13:34.815769 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:34.815746 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-229tg" event={"ID":"0c82a3c9-4883-4803-9762-f008f3dfea21","Type":"ContainerStarted","Data":"c06b622a263aeef146a70789229d354dfc5bd81fb158b7414668aab99948af3d"} Jul 31 14:13:39.832130 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:39.832097 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-229tg" event={"ID":"0c82a3c9-4883-4803-9762-f008f3dfea21","Type":"ContainerStarted","Data":"459300b61c1fc0394fe1ceef27bfcc105545424a402883b86bebede540b706b6"} Jul 31 14:13:39.832640 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:39.832196 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:13:39.833603 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:39.833578 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lsrr4" event={"ID":"72dad2d5-e7ca-4509-8619-e30eb609e348","Type":"ContainerStarted","Data":"ddf80e0f52484676bd8bbc6bfe24aac7c3c24b95d67d4016889843906625869c"} Jul 31 14:13:39.845320 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:39.845276 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-229tg" podStartSLOduration=67.114089491 podStartE2EDuration="1m11.845263152s" totalImagesPullingTime="4.731173661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:39.844704401 +0000 UTC m=+71.881527468" watchObservedRunningTime="2026-07-31 14:13:39.845263152 +0000 UTC m=+71.882086196" Jul 31 14:13:39.857265 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:13:39.857224 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lsrr4" podStartSLOduration=67.110892442 podStartE2EDuration="1m11.857209268s" totalImagesPullingTime="4.746316826s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:39.85700344 +0000 UTC m=+71.893826507" watchObservedRunningTime="2026-07-31 14:13:39.857209268 +0000 UTC m=+71.894032335" Jul 31 14:14:06.717989 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:14:06.717858 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:14:06.718422 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:14:06.718016 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:14:06.718422 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:14:06.718108 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls podName:7655bcd6-8ab3-4f31-bd58-7ebfef6f840b nodeName:}" failed. No retries permitted until 2026-07-31 14:15:10.718085122 +0000 UTC m=+162.754908170 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls") pod "dns-default-6lm6j" (UID: "7655bcd6-8ab3-4f31-bd58-7ebfef6f840b") : secret "dns-default-metrics-tls" not found Jul 31 14:14:10.838747 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:14:10.838713 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-229tg" Jul 31 14:14:38.131270 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:14:38.131232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs\") pod \"network-metrics-daemon-skd9g\" (UID: \"78e66929-08a3-4aa1-8a96-740f8f8eb7c8\") " pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:14:38.131657 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:14:38.131366 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 31 14:14:38.131657 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:14:38.131465 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs podName:78e66929-08a3-4aa1-8a96-740f8f8eb7c8 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:40.131448311 +0000 UTC m=+252.168271356 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs") pod "network-metrics-daemon-skd9g" (UID: "78e66929-08a3-4aa1-8a96-740f8f8eb7c8") : secret "metrics-daemon-secret" not found Jul 31 14:15:05.817272 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:15:05.817232 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6lm6j" podUID="7655bcd6-8ab3-4f31-bd58-7ebfef6f840b" Jul 31 14:15:06.084532 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:15:06.084450 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6lm6j" Jul 31 14:15:06.447588 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:15:06.447557 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:15:10.753024 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:15:10.752991 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:15:10.753430 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:15:10.753109 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:15:10.753430 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:15:10.753171 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls podName:7655bcd6-8ab3-4f31-bd58-7ebfef6f840b nodeName:}" failed. No retries permitted until 2026-07-31 14:17:12.7531573 +0000 UTC m=+284.789980345 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls") pod "dns-default-6lm6j" (UID: "7655bcd6-8ab3-4f31-bd58-7ebfef6f840b") : secret "dns-default-metrics-tls" not found Jul 31 14:15:17.118160 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:15:17.118123 2577 generic.go:356] "Generic (PLEG): container finished" podID="1d929dcc-18d9-4928-a93e-bd8fdb77100e" containerID="4905c282265edf760ef251becf7ef5af6ec29a07489ae5f4b03861c34d569068" exitCode=255 Jul 31 14:15:17.118553 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:15:17.118176 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk" event={"ID":"1d929dcc-18d9-4928-a93e-bd8fdb77100e","Type":"ContainerDied","Data":"4905c282265edf760ef251becf7ef5af6ec29a07489ae5f4b03861c34d569068"} Jul 31 14:15:17.118553 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:15:17.118443 2577 scope.go:117] "RemoveContainer" containerID="4905c282265edf760ef251becf7ef5af6ec29a07489ae5f4b03861c34d569068" Jul 31 14:15:18.123812 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:15:18.123776 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk" event={"ID":"1d929dcc-18d9-4928-a93e-bd8fdb77100e","Type":"ContainerStarted","Data":"09ada5f464e4310efab4c5330213fcd207fa148b83f61756180907d54785d925"} Jul 31 14:15:21.424404 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:15:21.424319 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:16:35.174438 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.174405 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fr5dq"] Jul 31 14:16:35.176529 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.176511 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:35.179791 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.179771 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 31 14:16:35.180136 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.180116 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 31 14:16:35.180226 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.180165 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t7s2z\"" Jul 31 14:16:35.180226 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.180193 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 31 14:16:35.180636 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.180612 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 31 14:16:35.186471 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.186449 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fr5dq"] Jul 31 14:16:35.215963 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.215940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7143a362-9b64-4ff5-aba1-40effb7058c0-crio-socket\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:35.216074 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.215971 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfpwz\" (UniqueName: \"kubernetes.io/projected/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-api-access-rfpwz\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:35.216074 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.215991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7143a362-9b64-4ff5-aba1-40effb7058c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:35.216074 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.216019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7143a362-9b64-4ff5-aba1-40effb7058c0-data-volume\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:35.216172 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.216085 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:35.316659 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.316629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7143a362-9b64-4ff5-aba1-40effb7058c0-crio-socket\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:35.316822 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.316665 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfpwz\" (UniqueName: \"kubernetes.io/projected/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-api-access-rfpwz\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:35.316822 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.316687 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7143a362-9b64-4ff5-aba1-40effb7058c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:35.316822 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.316715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7143a362-9b64-4ff5-aba1-40effb7058c0-data-volume\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:35.316822 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.316748 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:35.316822 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.316771 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7143a362-9b64-4ff5-aba1-40effb7058c0-crio-socket\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:35.316822 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:35.316811 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 14:16:35.317087 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:35.316882 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 14:16:35.317087 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:35.316897 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7143a362-9b64-4ff5-aba1-40effb7058c0-insights-runtime-extractor-tls podName:7143a362-9b64-4ff5-aba1-40effb7058c0 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:35.816876675 +0000 UTC m=+247.853699726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7143a362-9b64-4ff5-aba1-40effb7058c0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fr5dq" (UID: "7143a362-9b64-4ff5-aba1-40effb7058c0") : secret "insights-runtime-extractor-tls" not found Jul 31 14:16:35.317087 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:35.316957 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-rbac-proxy-cm podName:7143a362-9b64-4ff5-aba1-40effb7058c0 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:35.816938722 +0000 UTC m=+247.853761768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fr5dq" (UID: "7143a362-9b64-4ff5-aba1-40effb7058c0") : configmap "kube-rbac-proxy" not found Jul 31 14:16:35.317248 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.317179 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7143a362-9b64-4ff5-aba1-40effb7058c0-data-volume\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:35.324133 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.324107 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfpwz\" (UniqueName: \"kubernetes.io/projected/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-api-access-rfpwz\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:35.820524 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.820480 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:35.820726 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:35.820560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7143a362-9b64-4ff5-aba1-40effb7058c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:35.820726 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:35.820608 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 14:16:35.820726 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:35.820651 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 14:16:35.820726 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:35.820671 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-rbac-proxy-cm podName:7143a362-9b64-4ff5-aba1-40effb7058c0 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:36.82065481 +0000 UTC m=+248.857477855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fr5dq" (UID: "7143a362-9b64-4ff5-aba1-40effb7058c0") : configmap "kube-rbac-proxy" not found Jul 31 14:16:35.820726 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:35.820699 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7143a362-9b64-4ff5-aba1-40effb7058c0-insights-runtime-extractor-tls podName:7143a362-9b64-4ff5-aba1-40effb7058c0 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:36.820688509 +0000 UTC m=+248.857511554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7143a362-9b64-4ff5-aba1-40effb7058c0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fr5dq" (UID: "7143a362-9b64-4ff5-aba1-40effb7058c0") : secret "insights-runtime-extractor-tls" not found Jul 31 14:16:36.827636 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:36.827594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:36.828049 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:36.827664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7143a362-9b64-4ff5-aba1-40effb7058c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:36.828049 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:36.827716 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 14:16:36.828049 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:36.827756 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 14:16:36.828049 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:36.827785 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-rbac-proxy-cm podName:7143a362-9b64-4ff5-aba1-40effb7058c0 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:38.827767216 +0000 UTC m=+250.864590261 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fr5dq" (UID: "7143a362-9b64-4ff5-aba1-40effb7058c0") : configmap "kube-rbac-proxy" not found Jul 31 14:16:36.828049 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:36.827800 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7143a362-9b64-4ff5-aba1-40effb7058c0-insights-runtime-extractor-tls podName:7143a362-9b64-4ff5-aba1-40effb7058c0 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:38.827794061 +0000 UTC m=+250.864617105 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7143a362-9b64-4ff5-aba1-40effb7058c0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fr5dq" (UID: "7143a362-9b64-4ff5-aba1-40effb7058c0") : secret "insights-runtime-extractor-tls" not found Jul 31 14:16:37.139485 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:37.139458 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f2v62_80dc0e3d-60fa-46e3-9e00-20eefb9986eb/dns-node-resolver/0.log" Jul 31 14:16:38.338771 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:38.338744 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qltd5_0fd2b5b1-f721-4a2e-8c19-1c220634d162/node-ca/0.log" Jul 31 14:16:38.740080 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:38.740053 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8l2tx_43e35457-db4f-45d4-a5ac-dd16537435d6/serve-healthcheck-canary/0.log" Jul 31 14:16:38.844979 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:38.844941 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7143a362-9b64-4ff5-aba1-40effb7058c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:38.845140 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:38.845014 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:38.845140 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:38.845084 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 14:16:38.845223 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:38.845147 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7143a362-9b64-4ff5-aba1-40effb7058c0-insights-runtime-extractor-tls podName:7143a362-9b64-4ff5-aba1-40effb7058c0 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:42.845132354 +0000 UTC m=+254.881955402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7143a362-9b64-4ff5-aba1-40effb7058c0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fr5dq" (UID: "7143a362-9b64-4ff5-aba1-40effb7058c0") : secret "insights-runtime-extractor-tls" not found Jul 31 14:16:38.845223 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:38.845092 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 14:16:38.845223 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:38.845194 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-rbac-proxy-cm podName:7143a362-9b64-4ff5-aba1-40effb7058c0 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:42.845183016 +0000 UTC m=+254.882006062 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fr5dq" (UID: "7143a362-9b64-4ff5-aba1-40effb7058c0") : configmap "kube-rbac-proxy" not found Jul 31 14:16:40.155489 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:40.155455 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs\") pod \"network-metrics-daemon-skd9g\" (UID: \"78e66929-08a3-4aa1-8a96-740f8f8eb7c8\") " pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:16:40.155879 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:40.155587 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 31 14:16:40.155879 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:40.155660 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs podName:78e66929-08a3-4aa1-8a96-740f8f8eb7c8 nodeName:}" failed. No retries permitted until 2026-07-31 14:18:42.155631831 +0000 UTC m=+374.192454885 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs") pod "network-metrics-daemon-skd9g" (UID: "78e66929-08a3-4aa1-8a96-740f8f8eb7c8") : secret "metrics-daemon-secret" not found Jul 31 14:16:42.874450 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:42.874419 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:42.874869 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:42.874478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7143a362-9b64-4ff5-aba1-40effb7058c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:42.874869 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:42.874535 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 14:16:42.874869 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:42.874563 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 14:16:42.874869 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:42.874596 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-rbac-proxy-cm podName:7143a362-9b64-4ff5-aba1-40effb7058c0 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:50.87458031 +0000 UTC m=+262.911403356 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fr5dq" (UID: "7143a362-9b64-4ff5-aba1-40effb7058c0") : configmap "kube-rbac-proxy" not found Jul 31 14:16:42.874869 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:16:42.874611 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7143a362-9b64-4ff5-aba1-40effb7058c0-insights-runtime-extractor-tls podName:7143a362-9b64-4ff5-aba1-40effb7058c0 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:50.874604966 +0000 UTC m=+262.911428011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7143a362-9b64-4ff5-aba1-40effb7058c0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fr5dq" (UID: "7143a362-9b64-4ff5-aba1-40effb7058c0") : secret "insights-runtime-extractor-tls" not found Jul 31 14:16:50.929968 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:50.929866 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:50.929968 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:50.929959 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7143a362-9b64-4ff5-aba1-40effb7058c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:50.930380 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:50.930356 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7143a362-9b64-4ff5-aba1-40effb7058c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:50.932139 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:50.932118 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7143a362-9b64-4ff5-aba1-40effb7058c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fr5dq\" (UID: \"7143a362-9b64-4ff5-aba1-40effb7058c0\") " pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:51.085064 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:51.085018 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fr5dq" Jul 31 14:16:51.205661 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:51.205590 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fr5dq"] Jul 31 14:16:51.208589 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:16:51.208565 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7143a362_9b64_4ff5_aba1_40effb7058c0.slice/crio-b2812636ad8adb1638b6562a83b0b4136af6f56ef539599d6c93aa8d209391d6 WatchSource:0}: Error finding container b2812636ad8adb1638b6562a83b0b4136af6f56ef539599d6c93aa8d209391d6: Status 404 returned error can't find the container with id b2812636ad8adb1638b6562a83b0b4136af6f56ef539599d6c93aa8d209391d6 Jul 31 14:16:51.384539 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:51.384499 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fr5dq" event={"ID":"7143a362-9b64-4ff5-aba1-40effb7058c0","Type":"ContainerStarted","Data":"152256d1d1b301160ca6b1898cf8ccfa4f89186e3066444008d8fe0e542d507d"} Jul 31 14:16:51.384699 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:51.384556 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fr5dq" event={"ID":"7143a362-9b64-4ff5-aba1-40effb7058c0","Type":"ContainerStarted","Data":"b2812636ad8adb1638b6562a83b0b4136af6f56ef539599d6c93aa8d209391d6"} Jul 31 14:16:52.388601 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:52.388567 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fr5dq" event={"ID":"7143a362-9b64-4ff5-aba1-40effb7058c0","Type":"ContainerStarted","Data":"767e4f9c9777022d2ff140314f2bfe3e6cb60578bd43f331a7cbfcad8a862900"} Jul 31 14:16:53.393631 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:53.393602 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fr5dq" event={"ID":"7143a362-9b64-4ff5-aba1-40effb7058c0","Type":"ContainerStarted","Data":"9a462f2492276fb23a79cfc8b477fd373bbd33188245cd11c4f4f5b09f67a0ec"} Jul 31 14:16:53.408890 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:16:53.408829 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fr5dq" podStartSLOduration=16.414246656 podStartE2EDuration="18.408814466s" totalImagesPullingTime="1.99456781s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:16:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:16:53.407667325 +0000 UTC m=+265.444490391" watchObservedRunningTime="2026-07-31 14:16:53.408814466 +0000 UTC m=+265.445637533" Jul 31 14:17:08.059427 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.059389 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6"] Jul 31 14:17:08.062810 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.062796 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" Jul 31 14:17:08.065024 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.064992 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 31 14:17:08.065160 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.064992 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7gffh\"" Jul 31 14:17:08.065160 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.064997 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 31 14:17:08.065367 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.065347 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 31 14:17:08.065449 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.065394 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 31 14:17:08.065449 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.065417 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 31 14:17:08.071534 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.071514 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6"] Jul 31 14:17:08.077405 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.077388 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l2kdv"] Jul 31 14:17:08.080410 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.080394 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.082107 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.082088 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 31 14:17:08.082179 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.082092 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 31 14:17:08.082179 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.082171 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 31 14:17:08.082281 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.082246 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zn2sg\"" Jul 31 14:17:08.147483 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.147454 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f0c43be-78de-4de3-99a1-e1001f6208c8-root\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.147483 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.147483 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtwsz\" (UniqueName: \"kubernetes.io/projected/9f0c43be-78de-4de3-99a1-e1001f6208c8-kube-api-access-mtwsz\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.147718 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.147502 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfvnf\" (UniqueName: \"kubernetes.io/projected/ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd-kube-api-access-zfvnf\") pod \"openshift-state-metrics-6bb4d7b4dd-wlzs6\" (UID: \"ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" Jul 31 14:17:08.147718 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.147533 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f0c43be-78de-4de3-99a1-e1001f6208c8-node-exporter-tls\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.147718 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.147591 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-wlzs6\" (UID: \"ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" Jul 31 14:17:08.147718 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.147634 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f0c43be-78de-4de3-99a1-e1001f6208c8-metrics-client-ca\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.147718 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.147675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f0c43be-78de-4de3-99a1-e1001f6208c8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.147718 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.147708 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-wlzs6\" (UID: \"ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" Jul 31 14:17:08.147969 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.147760 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f0c43be-78de-4de3-99a1-e1001f6208c8-node-exporter-accelerators-collector-config\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.147969 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.147821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-wlzs6\" (UID: \"ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" Jul 31 14:17:08.147969 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.147870 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f0c43be-78de-4de3-99a1-e1001f6208c8-sys\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.147969 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.147887 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f0c43be-78de-4de3-99a1-e1001f6208c8-node-exporter-textfile\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.147969 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.147904 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f0c43be-78de-4de3-99a1-e1001f6208c8-node-exporter-wtmp\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.249137 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.249093 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f0c43be-78de-4de3-99a1-e1001f6208c8-sys\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.249137 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.249138 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f0c43be-78de-4de3-99a1-e1001f6208c8-node-exporter-textfile\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.249372 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.249155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f0c43be-78de-4de3-99a1-e1001f6208c8-node-exporter-wtmp\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.249372 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.249192 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f0c43be-78de-4de3-99a1-e1001f6208c8-root\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.249372 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.249213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtwsz\" (UniqueName: \"kubernetes.io/projected/9f0c43be-78de-4de3-99a1-e1001f6208c8-kube-api-access-mtwsz\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.249372 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.249236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvnf\" (UniqueName: \"kubernetes.io/projected/ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd-kube-api-access-zfvnf\") pod \"openshift-state-metrics-6bb4d7b4dd-wlzs6\" (UID: \"ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" Jul 31 14:17:08.249372 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.249212 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f0c43be-78de-4de3-99a1-e1001f6208c8-sys\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.249372 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.249261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f0c43be-78de-4de3-99a1-e1001f6208c8-node-exporter-tls\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.249372 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.249244 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f0c43be-78de-4de3-99a1-e1001f6208c8-root\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.249372 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.249323 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f0c43be-78de-4de3-99a1-e1001f6208c8-node-exporter-wtmp\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.249372 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.249369 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-wlzs6\" (UID: \"ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" Jul 31 14:17:08.249799 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.249399 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f0c43be-78de-4de3-99a1-e1001f6208c8-metrics-client-ca\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.249799 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.249437 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f0c43be-78de-4de3-99a1-e1001f6208c8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.249799 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.249468 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-wlzs6\" (UID: \"ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" Jul 31 14:17:08.249799 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.249503 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f0c43be-78de-4de3-99a1-e1001f6208c8-node-exporter-textfile\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.249799 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.249531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f0c43be-78de-4de3-99a1-e1001f6208c8-node-exporter-accelerators-collector-config\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.249799 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.249574 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-wlzs6\" (UID: \"ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" Jul 31 14:17:08.250206 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.250178 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f0c43be-78de-4de3-99a1-e1001f6208c8-node-exporter-accelerators-collector-config\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.250305 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.250178 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f0c43be-78de-4de3-99a1-e1001f6208c8-metrics-client-ca\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.250563 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.250539 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-wlzs6\" (UID: \"ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" Jul 31 14:17:08.251912 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.251888 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f0c43be-78de-4de3-99a1-e1001f6208c8-node-exporter-tls\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.251912 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.251908 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-wlzs6\" (UID: \"ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" Jul 31 14:17:08.252141 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.252116 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f0c43be-78de-4de3-99a1-e1001f6208c8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.252187 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.252172 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-wlzs6\" (UID: \"ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" Jul 31 14:17:08.258321 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.258302 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtwsz\" (UniqueName: \"kubernetes.io/projected/9f0c43be-78de-4de3-99a1-e1001f6208c8-kube-api-access-mtwsz\") pod \"node-exporter-l2kdv\" (UID: \"9f0c43be-78de-4de3-99a1-e1001f6208c8\") " pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.258416 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.258342 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfvnf\" (UniqueName: \"kubernetes.io/projected/ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd-kube-api-access-zfvnf\") pod \"openshift-state-metrics-6bb4d7b4dd-wlzs6\" (UID: \"ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" Jul 31 14:17:08.372334 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.372246 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" Jul 31 14:17:08.392686 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.392648 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l2kdv" Jul 31 14:17:08.405000 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:17:08.404973 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0c43be_78de_4de3_99a1_e1001f6208c8.slice/crio-ba2e4ae93c2509bfb0391c5b0bd0977ea7a54b40f052ce512482563565916c81 WatchSource:0}: Error finding container ba2e4ae93c2509bfb0391c5b0bd0977ea7a54b40f052ce512482563565916c81: Status 404 returned error can't find the container with id ba2e4ae93c2509bfb0391c5b0bd0977ea7a54b40f052ce512482563565916c81 Jul 31 14:17:08.442023 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.441977 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2kdv" event={"ID":"9f0c43be-78de-4de3-99a1-e1001f6208c8","Type":"ContainerStarted","Data":"ba2e4ae93c2509bfb0391c5b0bd0977ea7a54b40f052ce512482563565916c81"} Jul 31 14:17:08.501625 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:08.501590 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6"] Jul 31 14:17:08.505425 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:17:08.505399 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8ae95a_3a6e_476a_aafb_78b54aa6ddcd.slice/crio-21b49378968a960739c634d087d289a5248793836fb8b92b5c9ba5c68041cddc WatchSource:0}: Error finding container 21b49378968a960739c634d087d289a5248793836fb8b92b5c9ba5c68041cddc: Status 404 returned error can't find the container with id 21b49378968a960739c634d087d289a5248793836fb8b92b5c9ba5c68041cddc Jul 31 14:17:09.085769 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:17:09.085705 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6lm6j" podUID="7655bcd6-8ab3-4f31-bd58-7ebfef6f840b" Jul 31 14:17:09.447145 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:09.447114 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6lm6j" Jul 31 14:17:09.447145 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:09.447128 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" event={"ID":"ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd","Type":"ContainerStarted","Data":"7e6219928f022b7ed703aa7eac9660cb34b0fcd336e82e0cce1c1b2fa26f9692"} Jul 31 14:17:09.447408 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:09.447195 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" event={"ID":"ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd","Type":"ContainerStarted","Data":"88889a82c0f746788fe76e8d3b38cd706b3655cb9419925425a7b22a82ad547d"} Jul 31 14:17:09.447408 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:09.447227 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" event={"ID":"ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd","Type":"ContainerStarted","Data":"21b49378968a960739c634d087d289a5248793836fb8b92b5c9ba5c68041cddc"} Jul 31 14:17:10.453559 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:10.453523 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" event={"ID":"ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd","Type":"ContainerStarted","Data":"b6ac65eee82cb8871b8b3b1bec78cc999e662a7716e0b7d64888061ab0ed0364"} Jul 31 14:17:10.469119 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:10.469070 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wlzs6" podStartSLOduration=1.300068845 podStartE2EDuration="2.46905102s" totalImagesPullingTime="1.168982175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:10.467813463 +0000 UTC m=+282.504636531" watchObservedRunningTime="2026-07-31 14:17:10.46905102 +0000 UTC m=+282.505874089" Jul 31 14:17:11.458772 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:11.458728 2577 generic.go:356] "Generic (PLEG): container finished" podID="9f0c43be-78de-4de3-99a1-e1001f6208c8" containerID="c3c9008b8c6d215f4816bd7a0379c64801c7db3d93d7f17b0249477b627fa871" exitCode=0 Jul 31 14:17:11.459281 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:11.458822 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2kdv" event={"ID":"9f0c43be-78de-4de3-99a1-e1001f6208c8","Type":"ContainerDied","Data":"c3c9008b8c6d215f4816bd7a0379c64801c7db3d93d7f17b0249477b627fa871"} Jul 31 14:17:12.376854 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.376807 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6b49c5d89f-kbzr2"] Jul 31 14:17:12.381617 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.381588 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.383987 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.383962 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 31 14:17:12.384128 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.383961 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 31 14:17:12.384128 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.384049 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 31 14:17:12.384128 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.384067 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b8t0o5ds32skm\"" Jul 31 14:17:12.384128 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.384091 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 31 14:17:12.384128 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.384107 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mtxk5\"" Jul 31 14:17:12.387558 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.387530 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b49c5d89f-kbzr2"] Jul 31 14:17:12.463501 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.463464 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2kdv" event={"ID":"9f0c43be-78de-4de3-99a1-e1001f6208c8","Type":"ContainerStarted","Data":"a5ea1252976479544dac5752381e1b8639af378a63d22f32d6e7fd06a652bf2a"} Jul 31 14:17:12.463883 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.463517 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2kdv" event={"ID":"9f0c43be-78de-4de3-99a1-e1001f6208c8","Type":"ContainerStarted","Data":"42865ff12a04cbb09a460235315c07780955f21f93ad77d4e7e4465123035e13"} Jul 31 14:17:12.478939 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.478864 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l2kdv" podStartSLOduration=2.181199584 podStartE2EDuration="4.478822965s" totalImagesPullingTime="2.235468394s" totalInitContainerRuntime="62.154987ms" isStatefulPod=true podCreationTimestamp="2026-07-31 14:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:12.478394388 +0000 UTC m=+284.515217456" watchObservedRunningTime="2026-07-31 14:17:12.478822965 +0000 UTC m=+284.515646032" Jul 31 14:17:12.483710 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.483690 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffpjj\" (UniqueName: \"kubernetes.io/projected/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-kube-api-access-ffpjj\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.483783 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.483721 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-secret-metrics-server-tls\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.483783 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.483750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-metrics-server-audit-profiles\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.483874 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.483837 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-secret-metrics-server-client-certs\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.483957 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.483942 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.483993 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.483977 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-audit-log\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.484026 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.483999 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-client-ca-bundle\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.584916 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.584879 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.585065 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.584945 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-audit-log\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.585065 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.584972 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-client-ca-bundle\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.585065 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.584995 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffpjj\" (UniqueName: \"kubernetes.io/projected/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-kube-api-access-ffpjj\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.585065 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.585024 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-secret-metrics-server-tls\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.585264 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.585068 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-metrics-server-audit-profiles\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.585264 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.585121 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-secret-metrics-server-client-certs\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.585427 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.585406 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-audit-log\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.585700 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.585675 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.586285 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.586255 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-metrics-server-audit-profiles\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.587561 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.587534 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-client-ca-bundle\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.587644 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.587617 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-secret-metrics-server-client-certs\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.587683 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.587642 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-secret-metrics-server-tls\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.592030 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.592014 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffpjj\" (UniqueName: \"kubernetes.io/projected/3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4-kube-api-access-ffpjj\") pod \"metrics-server-6b49c5d89f-kbzr2\" (UID: \"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4\") " pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.691131 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.691093 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:12.786708 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.786676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:17:12.789511 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.789482 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7655bcd6-8ab3-4f31-bd58-7ebfef6f840b-metrics-tls\") pod \"dns-default-6lm6j\" (UID: \"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b\") " pod="openshift-dns/dns-default-6lm6j" Jul 31 14:17:12.821828 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:12.821795 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b49c5d89f-kbzr2"] Jul 31 14:17:12.825457 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:17:12.825430 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3df8bb_e9bc_42ff_af0c_efeef3d9b0c4.slice/crio-8e3f3e0c0c62c0e9b9ce29751028d86bc34a772b0584183ca3027adf4d53f1f4 WatchSource:0}: Error finding container 8e3f3e0c0c62c0e9b9ce29751028d86bc34a772b0584183ca3027adf4d53f1f4: Status 404 returned error can't find the container with id 8e3f3e0c0c62c0e9b9ce29751028d86bc34a772b0584183ca3027adf4d53f1f4 Jul 31 14:17:13.049610 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:13.049528 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p9vdf\"" Jul 31 14:17:13.058882 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:13.058835 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6lm6j" Jul 31 14:17:13.177947 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:13.177916 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6lm6j"] Jul 31 14:17:13.180936 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:17:13.180904 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7655bcd6_8ab3_4f31_bd58_7ebfef6f840b.slice/crio-51de3a256274e2cb1f941007e7f6b9e90e8ca014db02c44828f1eb14ac62c7ff WatchSource:0}: Error finding container 51de3a256274e2cb1f941007e7f6b9e90e8ca014db02c44828f1eb14ac62c7ff: Status 404 returned error can't find the container with id 51de3a256274e2cb1f941007e7f6b9e90e8ca014db02c44828f1eb14ac62c7ff Jul 31 14:17:13.467826 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:13.467792 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6lm6j" event={"ID":"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b","Type":"ContainerStarted","Data":"51de3a256274e2cb1f941007e7f6b9e90e8ca014db02c44828f1eb14ac62c7ff"} Jul 31 14:17:13.468886 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:13.468864 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" event={"ID":"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4","Type":"ContainerStarted","Data":"8e3f3e0c0c62c0e9b9ce29751028d86bc34a772b0584183ca3027adf4d53f1f4"} Jul 31 14:17:14.243948 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.243914 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 14:17:14.247599 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.247579 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.249729 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.249708 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 31 14:17:14.249820 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.249761 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 31 14:17:14.249820 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.249772 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 31 14:17:14.249999 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.249982 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-n4nzf\"" Jul 31 14:17:14.250065 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.250022 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 31 14:17:14.250065 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.250032 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 31 14:17:14.250166 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.250024 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 31 14:17:14.250274 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.250257 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 31 14:17:14.250342 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.250317 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 31 14:17:14.250401 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.250383 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 31 14:17:14.250508 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.250425 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 31 14:17:14.250570 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.250522 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 31 14:17:14.250741 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.250726 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 31 14:17:14.250809 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.250746 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 31 14:17:14.250809 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.250752 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-eb5sfffspau00\"" Jul 31 14:17:14.253017 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.252998 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 31 14:17:14.263089 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.263072 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 14:17:14.298169 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298147 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298266 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298173 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298266 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298191 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298266 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298216 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298266 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298231 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298421 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298268 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298421 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298328 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298421 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298412 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0bdb99b-3d57-46c2-86b9-ac08c7333142-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298507 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298449 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298507 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298471 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r76n\" (UniqueName: \"kubernetes.io/projected/c0bdb99b-3d57-46c2-86b9-ac08c7333142-kube-api-access-9r76n\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298507 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298488 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-config\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298613 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298515 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298613 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298531 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298613 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298547 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298613 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298573 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298613 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298591 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298613 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298889 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298889 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298709 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0bdb99b-3d57-46c2-86b9-ac08c7333142-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.298889 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.298744 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.399483 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.399451 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.399483 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.399488 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0bdb99b-3d57-46c2-86b9-ac08c7333142-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.399708 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.399510 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.399708 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.399545 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.399708 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.399572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.399708 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.399601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.399708 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.399644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.399708 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.399666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.399708 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.399690 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.400051 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.399715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.400051 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.399747 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0bdb99b-3d57-46c2-86b9-ac08c7333142-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.400051 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.399778 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.400051 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.399810 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r76n\" (UniqueName: \"kubernetes.io/projected/c0bdb99b-3d57-46c2-86b9-ac08c7333142-kube-api-access-9r76n\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.400051 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.399835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-config\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.400051 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.399903 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.400051 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.399936 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.400051 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.399959 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.400051 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.399992 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.400051 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.400021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.400051 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.400043 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.400519 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.400085 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.400519 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.400467 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.400623 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.400604 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.401312 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.401282 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.406804 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.404040 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.406804 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.403959 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.406804 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.404577 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.406804 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.404652 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.406804 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.404673 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.406804 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.404696 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.406804 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.405207 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.406804 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.405998 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.406804 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.406430 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.406804 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.406676 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-config\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.407352 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.406872 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.407352 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.406937 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.407352 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.406944 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0bdb99b-3d57-46c2-86b9-ac08c7333142-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.407592 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.407564 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0bdb99b-3d57-46c2-86b9-ac08c7333142-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.407734 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.407707 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.409659 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.409641 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r76n\" (UniqueName: \"kubernetes.io/projected/c0bdb99b-3d57-46c2-86b9-ac08c7333142-kube-api-access-9r76n\") pod \"prometheus-k8s-0\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.557754 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.557671 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:14.941800 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:14.941773 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 14:17:15.480380 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:15.480345 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6lm6j" event={"ID":"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b","Type":"ContainerStarted","Data":"26b6f3453bae6627e65867cc793b1c2d4e750195e90b4227709c0cb6bbf0a0d3"} Jul 31 14:17:15.480545 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:15.480402 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6lm6j" event={"ID":"7655bcd6-8ab3-4f31-bd58-7ebfef6f840b","Type":"ContainerStarted","Data":"6863c89c6497bb8162b00c14f0b7b2dbc358572897199c3f7c917669f848be52"} Jul 31 14:17:15.480545 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:15.480433 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6lm6j" Jul 31 14:17:15.482146 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:15.482113 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0bdb99b-3d57-46c2-86b9-ac08c7333142","Type":"ContainerStarted","Data":"faa2bdd8f48a876ba5d14e5b11d3bbba157e2fc8c2b294030c8d0c3c98eff646"} Jul 31 14:17:15.495729 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:15.495676 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6lm6j" podStartSLOduration=251.83182182 podStartE2EDuration="4m13.495659643s" totalImagesPullingTime="1.663837823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:13:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:15.494243786 +0000 UTC m=+287.531066853" watchObservedRunningTime="2026-07-31 14:17:15.495659643 +0000 UTC m=+287.532482714" Jul 31 14:17:16.486661 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:16.486626 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" event={"ID":"3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4","Type":"ContainerStarted","Data":"f0b978dc156afbbd3ce5c58287715e2a2780bcabc87afcc9943d19a74c944b1d"} Jul 31 14:17:16.488174 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:16.488147 2577 generic.go:356] "Generic (PLEG): container finished" podID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerID="cbcfe70f34c95eeb0c9847e061fecf412af974a7689516efab08675bd7010504" exitCode=0 Jul 31 14:17:16.488304 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:16.488221 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0bdb99b-3d57-46c2-86b9-ac08c7333142","Type":"ContainerDied","Data":"cbcfe70f34c95eeb0c9847e061fecf412af974a7689516efab08675bd7010504"} Jul 31 14:17:16.501409 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:16.501362 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" podStartSLOduration=1.2475891940000001 podStartE2EDuration="4.501343542s" totalImagesPullingTime="3.253754348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:16.501342664 +0000 UTC m=+288.538165732" watchObservedRunningTime="2026-07-31 14:17:16.501343542 +0000 UTC m=+288.538166612" Jul 31 14:17:17.494814 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:17.494728 2577 generic.go:356] "Generic (PLEG): container finished" podID="1d929dcc-18d9-4928-a93e-bd8fdb77100e" containerID="09ada5f464e4310efab4c5330213fcd207fa148b83f61756180907d54785d925" exitCode=255 Jul 31 14:17:17.494814 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:17.494800 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk" event={"ID":"1d929dcc-18d9-4928-a93e-bd8fdb77100e","Type":"ContainerDied","Data":"09ada5f464e4310efab4c5330213fcd207fa148b83f61756180907d54785d925"} Jul 31 14:17:17.495318 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:17.494869 2577 scope.go:117] "RemoveContainer" containerID="4905c282265edf760ef251becf7ef5af6ec29a07489ae5f4b03861c34d569068" Jul 31 14:17:17.495318 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:17.495307 2577 scope.go:117] "RemoveContainer" containerID="09ada5f464e4310efab4c5330213fcd207fa148b83f61756180907d54785d925" Jul 31 14:17:17.495525 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:17:17.495500 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-6fc587fd96-cqpjk_open-cluster-management-agent-addon(1d929dcc-18d9-4928-a93e-bd8fdb77100e)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk" podUID="1d929dcc-18d9-4928-a93e-bd8fdb77100e" Jul 31 14:17:20.452745 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:20.452707 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk" Jul 31 14:17:20.453211 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:20.453039 2577 scope.go:117] "RemoveContainer" containerID="09ada5f464e4310efab4c5330213fcd207fa148b83f61756180907d54785d925" Jul 31 14:17:20.453270 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:17:20.453225 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-6fc587fd96-cqpjk_open-cluster-management-agent-addon(1d929dcc-18d9-4928-a93e-bd8fdb77100e)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk" podUID="1d929dcc-18d9-4928-a93e-bd8fdb77100e" Jul 31 14:17:20.460214 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:20.460178 2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" podUID="5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 31 14:17:20.510434 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:20.510403 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0bdb99b-3d57-46c2-86b9-ac08c7333142","Type":"ContainerStarted","Data":"0dae4c41ec632cbedf1ab5148f57f95ba4572b1f5db730331c02df0a714c1b0c"} Jul 31 14:17:20.510560 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:20.510467 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0bdb99b-3d57-46c2-86b9-ac08c7333142","Type":"ContainerStarted","Data":"29fe7a4b117b48babd0babd8db6c8307636097ff85b317e7ba18684a1c7c5241"} Jul 31 14:17:21.517887 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:21.517834 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0bdb99b-3d57-46c2-86b9-ac08c7333142","Type":"ContainerStarted","Data":"6c9ecb454f0dc29798ff92115e7f016b5c5aa5f7884fd6f4e88f168a99de74ae"} Jul 31 14:17:22.524081 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:22.524044 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0bdb99b-3d57-46c2-86b9-ac08c7333142","Type":"ContainerStarted","Data":"edf2e11e8232a45f52fc53c66c13700b13c19943a443854abbf9da6d71861ec3"} Jul 31 14:17:22.524426 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:22.524119 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0bdb99b-3d57-46c2-86b9-ac08c7333142","Type":"ContainerStarted","Data":"ec6bc4ebc671b60033173d5063018b88935ca9a606d5a94e602eced349148d2b"} Jul 31 14:17:22.524426 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:22.524160 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0bdb99b-3d57-46c2-86b9-ac08c7333142","Type":"ContainerStarted","Data":"83d84f5f7dec7ffb1622a87467012d26bbdf7ba393d75f97c20934a0ab5f8811"} Jul 31 14:17:22.548667 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:22.548622 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.620988563 podStartE2EDuration="8.548608629s" totalImagesPullingTime="5.907978908s" totalInitContainerRuntime="19.641158ms" isStatefulPod=false podCreationTimestamp="2026-07-31 14:17:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:22.54719018 +0000 UTC m=+294.584013246" watchObservedRunningTime="2026-07-31 14:17:22.548608629 +0000 UTC m=+294.585431697" Jul 31 14:17:24.425192 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:17:24.425147 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-skd9g" podUID="78e66929-08a3-4aa1-8a96-740f8f8eb7c8" Jul 31 14:17:24.558578 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:24.558549 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:17:25.490726 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:25.490693 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6lm6j" Jul 31 14:17:28.313085 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:28.313053 2577 kubelet.go:1648] "Image garbage collection succeeded" Jul 31 14:17:30.459408 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:30.459370 2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" podUID="5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 31 14:17:32.691938 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:32.691903 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:32.692312 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:32.691960 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:33.425152 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:33.425119 2577 scope.go:117] "RemoveContainer" containerID="09ada5f464e4310efab4c5330213fcd207fa148b83f61756180907d54785d925" Jul 31 14:17:33.568411 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:33.568380 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc587fd96-cqpjk" event={"ID":"1d929dcc-18d9-4928-a93e-bd8fdb77100e","Type":"ContainerStarted","Data":"c4972fab52f265782347380fdb059a0b6824012289617f0355e5fea348b7aebd"} Jul 31 14:17:39.424126 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:39.424089 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:17:40.459862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:40.459810 2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" podUID="5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 31 14:17:40.460253 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:40.459893 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" Jul 31 14:17:40.460368 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:40.460338 2577 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"8d66ba99efde0ace7aa78277008ae36f9f9fca8ddabef7133035d142d91979b1"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 31 14:17:40.460405 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:40.460387 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" podUID="5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964" containerName="service-proxy" containerID="cri-o://8d66ba99efde0ace7aa78277008ae36f9f9fca8ddabef7133035d142d91979b1" gracePeriod=30 Jul 31 14:17:40.592217 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:40.592186 2577 generic.go:356] "Generic (PLEG): container finished" podID="5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964" containerID="8d66ba99efde0ace7aa78277008ae36f9f9fca8ddabef7133035d142d91979b1" exitCode=2 Jul 31 14:17:40.592340 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:40.592262 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" event={"ID":"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964","Type":"ContainerDied","Data":"8d66ba99efde0ace7aa78277008ae36f9f9fca8ddabef7133035d142d91979b1"} Jul 31 14:17:41.597866 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:41.597807 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-684c8745c8-rcz6z" event={"ID":"5feafd4d-47c7-4c7b-b6ec-eafbb6f6d964","Type":"ContainerStarted","Data":"16b036262f560e171259dd3fb2270acb8d6027df74d4f706184e4e9c9cf48ca8"} Jul 31 14:17:52.697211 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:52.697173 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:17:52.701113 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:17:52.701089 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6b49c5d89f-kbzr2" Jul 31 14:18:14.558586 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:14.558550 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:14.577271 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:14.577233 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:14.722766 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:14.722739 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:32.601483 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:32.601402 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 14:18:32.601940 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:32.601707 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="prometheus" containerID="cri-o://29fe7a4b117b48babd0babd8db6c8307636097ff85b317e7ba18684a1c7c5241" gracePeriod=600 Jul 31 14:18:32.601940 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:32.601746 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="kube-rbac-proxy" containerID="cri-o://ec6bc4ebc671b60033173d5063018b88935ca9a606d5a94e602eced349148d2b" gracePeriod=600 Jul 31 14:18:32.601940 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:32.601798 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="kube-rbac-proxy-thanos" containerID="cri-o://edf2e11e8232a45f52fc53c66c13700b13c19943a443854abbf9da6d71861ec3" gracePeriod=600 Jul 31 14:18:32.601940 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:32.601794 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="config-reloader" containerID="cri-o://0dae4c41ec632cbedf1ab5148f57f95ba4572b1f5db730331c02df0a714c1b0c" gracePeriod=600 Jul 31 14:18:32.601940 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:32.601746 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="thanos-sidecar" containerID="cri-o://6c9ecb454f0dc29798ff92115e7f016b5c5aa5f7884fd6f4e88f168a99de74ae" gracePeriod=600 Jul 31 14:18:32.601940 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:32.601800 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="kube-rbac-proxy-web" containerID="cri-o://83d84f5f7dec7ffb1622a87467012d26bbdf7ba393d75f97c20934a0ab5f8811" gracePeriod=600 Jul 31 14:18:32.782939 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:32.782907 2577 generic.go:356] "Generic (PLEG): container finished" podID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerID="edf2e11e8232a45f52fc53c66c13700b13c19943a443854abbf9da6d71861ec3" exitCode=0 Jul 31 14:18:32.782939 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:32.782931 2577 generic.go:356] "Generic (PLEG): container finished" podID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerID="ec6bc4ebc671b60033173d5063018b88935ca9a606d5a94e602eced349148d2b" exitCode=0 Jul 31 14:18:32.782939 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:32.782939 2577 generic.go:356] "Generic (PLEG): container finished" podID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerID="6c9ecb454f0dc29798ff92115e7f016b5c5aa5f7884fd6f4e88f168a99de74ae" exitCode=0 Jul 31 14:18:32.782939 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:32.782946 2577 generic.go:356] "Generic (PLEG): container finished" podID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerID="0dae4c41ec632cbedf1ab5148f57f95ba4572b1f5db730331c02df0a714c1b0c" exitCode=0 Jul 31 14:18:32.783204 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:32.782951 2577 generic.go:356] "Generic (PLEG): container finished" podID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerID="29fe7a4b117b48babd0babd8db6c8307636097ff85b317e7ba18684a1c7c5241" exitCode=0 Jul 31 14:18:32.783204 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:32.782985 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0bdb99b-3d57-46c2-86b9-ac08c7333142","Type":"ContainerDied","Data":"edf2e11e8232a45f52fc53c66c13700b13c19943a443854abbf9da6d71861ec3"} Jul 31 14:18:32.783204 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:32.783057 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0bdb99b-3d57-46c2-86b9-ac08c7333142","Type":"ContainerDied","Data":"ec6bc4ebc671b60033173d5063018b88935ca9a606d5a94e602eced349148d2b"} Jul 31 14:18:32.783204 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:32.783083 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0bdb99b-3d57-46c2-86b9-ac08c7333142","Type":"ContainerDied","Data":"6c9ecb454f0dc29798ff92115e7f016b5c5aa5f7884fd6f4e88f168a99de74ae"} Jul 31 14:18:32.783204 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:32.783108 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0bdb99b-3d57-46c2-86b9-ac08c7333142","Type":"ContainerDied","Data":"0dae4c41ec632cbedf1ab5148f57f95ba4572b1f5db730331c02df0a714c1b0c"} Jul 31 14:18:32.783204 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:32.783133 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0bdb99b-3d57-46c2-86b9-ac08c7333142","Type":"ContainerDied","Data":"29fe7a4b117b48babd0babd8db6c8307636097ff85b317e7ba18684a1c7c5241"} Jul 31 14:18:33.790529 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.790503 2577 generic.go:356] "Generic (PLEG): container finished" podID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerID="83d84f5f7dec7ffb1622a87467012d26bbdf7ba393d75f97c20934a0ab5f8811" exitCode=0 Jul 31 14:18:33.790901 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.790570 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0bdb99b-3d57-46c2-86b9-ac08c7333142","Type":"ContainerDied","Data":"83d84f5f7dec7ffb1622a87467012d26bbdf7ba393d75f97c20934a0ab5f8811"} Jul 31 14:18:33.846527 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.846505 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:33.945444 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945378 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0bdb99b-3d57-46c2-86b9-ac08c7333142-tls-assets\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.945444 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945415 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-thanos-prometheus-http-client-file\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.945639 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945447 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-rulefiles-0\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.945639 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945464 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-configmap-kubelet-serving-ca-bundle\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.945639 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945480 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-kube-rbac-proxy\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.945639 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945499 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.945639 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945514 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0bdb99b-3d57-46c2-86b9-ac08c7333142-config-out\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.945639 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945530 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-rulefiles-1\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.945639 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945549 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-db\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.945639 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945574 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r76n\" (UniqueName: \"kubernetes.io/projected/c0bdb99b-3d57-46c2-86b9-ac08c7333142-kube-api-access-9r76n\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.945639 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945598 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-rulefiles-2\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.945639 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945632 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.946138 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945663 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-configmap-serving-certs-ca-bundle\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.946138 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945695 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-web-config\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.946138 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945718 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-config\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.946138 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945736 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-metrics-client-certs\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.946138 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945764 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-trusted-ca-bundle\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.946138 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945791 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-prometheus-k8s-tls\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.946138 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945815 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-configmap-metrics-client-ca\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.946138 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.945856 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-grpc-tls\") pod \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\" (UID: \"c0bdb99b-3d57-46c2-86b9-ac08c7333142\") " Jul 31 14:18:33.946138 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.946025 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:18:33.946557 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.946140 2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:33.946557 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.946503 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:18:33.947788 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.946791 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:18:33.947788 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.946951 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:18:33.947788 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.947217 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:18:33.947788 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.947679 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:18:33.947788 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.947742 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:18:33.948156 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.947883 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:18:33.948271 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.948227 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:18:33.948527 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.948492 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bdb99b-3d57-46c2-86b9-ac08c7333142-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:18:33.950041 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.950014 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:18:33.950145 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.950131 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:18:33.950353 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.950276 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bdb99b-3d57-46c2-86b9-ac08c7333142-config-out" (OuterVolumeSpecName: "config-out") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:18:33.950597 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.950561 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:18:33.950597 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.950588 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-config" (OuterVolumeSpecName: "config") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:18:33.950754 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.950592 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bdb99b-3d57-46c2-86b9-ac08c7333142-kube-api-access-9r76n" (OuterVolumeSpecName: "kube-api-access-9r76n") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "kube-api-access-9r76n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:18:33.950905 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.950887 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:18:33.950989 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.950907 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:18:33.951451 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.951432 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:18:33.959297 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:33.959275 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-web-config" (OuterVolumeSpecName: "web-config") pod "c0bdb99b-3d57-46c2-86b9-ac08c7333142" (UID: "c0bdb99b-3d57-46c2-86b9-ac08c7333142"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:18:34.046879 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.046818 2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-kube-rbac-proxy\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.046879 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.046875 2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.046879 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.046886 2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0bdb99b-3d57-46c2-86b9-ac08c7333142-config-out\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.046879 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.046895 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.047108 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.046903 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-db\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.047108 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.046912 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r76n\" (UniqueName: \"kubernetes.io/projected/c0bdb99b-3d57-46c2-86b9-ac08c7333142-kube-api-access-9r76n\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.047108 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.046923 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.047108 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.046931 2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.047108 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.046940 2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.047108 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.046948 2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-web-config\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.047108 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.046956 2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-config\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.047108 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.046965 2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-metrics-client-certs\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.047108 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.046976 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.047108 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.046985 2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.047108 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.046993 2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-configmap-metrics-client-ca\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.047108 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.047001 2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-secret-grpc-tls\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.047108 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.047010 2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0bdb99b-3d57-46c2-86b9-ac08c7333142-tls-assets\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.047108 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.047019 2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0bdb99b-3d57-46c2-86b9-ac08c7333142-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.047108 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.047027 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0bdb99b-3d57-46c2-86b9-ac08c7333142-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:18:34.796621 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.796588 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0bdb99b-3d57-46c2-86b9-ac08c7333142","Type":"ContainerDied","Data":"faa2bdd8f48a876ba5d14e5b11d3bbba157e2fc8c2b294030c8d0c3c98eff646"} Jul 31 14:18:34.796991 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.796632 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.796991 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.796661 2577 scope.go:117] "RemoveContainer" containerID="edf2e11e8232a45f52fc53c66c13700b13c19943a443854abbf9da6d71861ec3" Jul 31 14:18:34.804156 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.804130 2577 scope.go:117] "RemoveContainer" containerID="ec6bc4ebc671b60033173d5063018b88935ca9a606d5a94e602eced349148d2b" Jul 31 14:18:34.810765 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.810746 2577 scope.go:117] "RemoveContainer" containerID="83d84f5f7dec7ffb1622a87467012d26bbdf7ba393d75f97c20934a0ab5f8811" Jul 31 14:18:34.815622 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.815599 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 14:18:34.817261 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.817246 2577 scope.go:117] "RemoveContainer" containerID="6c9ecb454f0dc29798ff92115e7f016b5c5aa5f7884fd6f4e88f168a99de74ae" Jul 31 14:18:34.821862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.821829 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 14:18:34.823583 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.823567 2577 scope.go:117] "RemoveContainer" containerID="0dae4c41ec632cbedf1ab5148f57f95ba4572b1f5db730331c02df0a714c1b0c" Jul 31 14:18:34.829559 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.829544 2577 scope.go:117] "RemoveContainer" containerID="29fe7a4b117b48babd0babd8db6c8307636097ff85b317e7ba18684a1c7c5241" Jul 31 14:18:34.836046 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.836030 2577 scope.go:117] "RemoveContainer" containerID="cbcfe70f34c95eeb0c9847e061fecf412af974a7689516efab08675bd7010504" Jul 31 14:18:34.885194 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885167 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 14:18:34.885407 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885395 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="config-reloader" Jul 31 14:18:34.885447 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885408 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="config-reloader" Jul 31 14:18:34.885447 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885418 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="kube-rbac-proxy" Jul 31 14:18:34.885447 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885424 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="kube-rbac-proxy" Jul 31 14:18:34.885447 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885435 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="kube-rbac-proxy-web" Jul 31 14:18:34.885447 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885440 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="kube-rbac-proxy-web" Jul 31 14:18:34.885587 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885450 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="prometheus" Jul 31 14:18:34.885587 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885456 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="prometheus" Jul 31 14:18:34.885587 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885465 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="thanos-sidecar" Jul 31 14:18:34.885587 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885472 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="thanos-sidecar" Jul 31 14:18:34.885587 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885478 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="kube-rbac-proxy-thanos" Jul 31 14:18:34.885587 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885484 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="kube-rbac-proxy-thanos" Jul 31 14:18:34.885587 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885492 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="init-config-reloader" Jul 31 14:18:34.885587 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885497 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="init-config-reloader" Jul 31 14:18:34.885587 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885556 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="kube-rbac-proxy" Jul 31 14:18:34.885587 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885565 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="thanos-sidecar" Jul 31 14:18:34.885587 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885573 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="prometheus" Jul 31 14:18:34.885587 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885578 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="config-reloader" Jul 31 14:18:34.885587 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885583 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="kube-rbac-proxy-web" Jul 31 14:18:34.885587 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.885588 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" containerName="kube-rbac-proxy-thanos" Jul 31 14:18:34.888562 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.888546 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.890753 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.890732 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 31 14:18:34.890871 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.890755 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 31 14:18:34.890871 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.890814 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 31 14:18:34.890871 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.890829 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 31 14:18:34.891055 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.891038 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 31 14:18:34.891105 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.891071 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 31 14:18:34.891254 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.891238 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 31 14:18:34.891631 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.891610 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-n4nzf\"" Jul 31 14:18:34.891703 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.891610 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 31 14:18:34.891703 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.891693 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-eb5sfffspau00\"" Jul 31 14:18:34.891871 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.891857 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 31 14:18:34.891871 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.891866 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 31 14:18:34.892126 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.892105 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 31 14:18:34.892284 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.892268 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 31 14:18:34.894818 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.894800 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 31 14:18:34.900729 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.900704 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 31 14:18:34.906268 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.906251 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 14:18:34.953801 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.953736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-config-out\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.953801 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.953767 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.953801 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.953788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.953976 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.953811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khgz6\" (UniqueName: \"kubernetes.io/projected/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-kube-api-access-khgz6\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.953976 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.953855 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.953976 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.953871 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.953976 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.953920 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.953976 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.953949 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.953976 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.953964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.954157 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.953990 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-config\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.954157 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.954005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-web-config\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.954157 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.954019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.954157 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.954042 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.954157 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.954056 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.954157 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.954073 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.954157 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.954087 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.954157 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.954106 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.954157 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.954125 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.954448 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.954167 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:34.954448 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:34.954198 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.055437 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055402 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.055437 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055436 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.055642 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055455 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.055642 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.055642 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055498 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-config-out\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.055642 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.055642 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055534 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.055642 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055552 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khgz6\" (UniqueName: \"kubernetes.io/projected/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-kube-api-access-khgz6\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.055642 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.055642 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.055642 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055603 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.055642 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055636 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.056121 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055658 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.056121 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055693 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-config\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.056121 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-web-config\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.056121 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055736 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.056121 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.056121 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055805 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.056121 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.056121 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.055883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.056526 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.056361 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.056526 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.056444 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.058646 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.058527 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.058646 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.058570 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-config\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.058821 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.058710 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.059974 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.059001 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.059974 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.059005 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.059974 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.059338 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.059974 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.059403 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.059974 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.059531 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.059974 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.059539 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.059974 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.059911 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.060397 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.060006 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.060397 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.060271 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.060614 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.060590 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.061079 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.061063 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.061347 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.061326 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-web-config\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.061760 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.061733 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-config-out\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.061972 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.061953 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.063551 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.063534 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khgz6\" (UniqueName: \"kubernetes.io/projected/c8cc199f-3ca5-45c4-8e3f-7a4768f80e24-kube-api-access-khgz6\") pod \"prometheus-k8s-0\" (UID: \"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.198440 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.198404 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:35.329415 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.329389 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 14:18:35.331828 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:18:35.331802 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8cc199f_3ca5_45c4_8e3f_7a4768f80e24.slice/crio-637c7ebb7c220c24ed4f34be323c54386b73f6e5bc4490ca938c59ffb370ac50 WatchSource:0}: Error finding container 637c7ebb7c220c24ed4f34be323c54386b73f6e5bc4490ca938c59ffb370ac50: Status 404 returned error can't find the container with id 637c7ebb7c220c24ed4f34be323c54386b73f6e5bc4490ca938c59ffb370ac50 Jul 31 14:18:35.802923 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.802890 2577 generic.go:356] "Generic (PLEG): container finished" podID="c8cc199f-3ca5-45c4-8e3f-7a4768f80e24" containerID="429565963e2696496c195fcecc5248b3cc52d75b7a7fbc96fdf1b924ae7d6eb9" exitCode=0 Jul 31 14:18:35.803276 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.802940 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24","Type":"ContainerDied","Data":"429565963e2696496c195fcecc5248b3cc52d75b7a7fbc96fdf1b924ae7d6eb9"} Jul 31 14:18:35.803276 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:35.802989 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24","Type":"ContainerStarted","Data":"637c7ebb7c220c24ed4f34be323c54386b73f6e5bc4490ca938c59ffb370ac50"} Jul 31 14:18:36.429249 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:36.429215 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0bdb99b-3d57-46c2-86b9-ac08c7333142" path="/var/lib/kubelet/pods/c0bdb99b-3d57-46c2-86b9-ac08c7333142/volumes" Jul 31 14:18:36.810195 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:36.810111 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24","Type":"ContainerStarted","Data":"bf023ab1d540c2d9c8a3cdf18e6b2a54843cd340bfde42d9a9c700457b662a55"} Jul 31 14:18:36.810195 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:36.810172 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24","Type":"ContainerStarted","Data":"1936d86561d08e077db081801ff3a4b803dbdeef5865ee27a09a1b3326d0cbac"} Jul 31 14:18:36.810575 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:36.810197 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24","Type":"ContainerStarted","Data":"b2de5bbf5244a38102f0fcb58d9c904f1a2d5c3fe3d50715facae9fc49db649e"} Jul 31 14:18:36.810575 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:36.810224 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24","Type":"ContainerStarted","Data":"be7340c62956f10c9268e06258ce96cd7ec50483c01da4b310cd01a7062e6b7e"} Jul 31 14:18:36.810575 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:36.810246 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24","Type":"ContainerStarted","Data":"8e814e84cc79d8ae28ab158df5b0cb4c81435d695d8f0cc3a38f68c8787f2550"} Jul 31 14:18:36.810575 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:36.810268 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8cc199f-3ca5-45c4-8e3f-7a4768f80e24","Type":"ContainerStarted","Data":"6adf6c5e2acb17dd429a14db305e586cb3c7e3f707e69519dbc3e10ee872fa38"} Jul 31 14:18:36.836865 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:36.835167 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.817111529 podStartE2EDuration="2.835147159s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.03563ms" isStatefulPod=false podCreationTimestamp="2026-07-31 14:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:18:36.833705654 +0000 UTC m=+368.870528734" watchObservedRunningTime="2026-07-31 14:18:36.835147159 +0000 UTC m=+368.871970227" Jul 31 14:18:40.198825 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:40.198791 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:18:42.210437 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:42.210386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs\") pod \"network-metrics-daemon-skd9g\" (UID: \"78e66929-08a3-4aa1-8a96-740f8f8eb7c8\") " pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:18:42.212780 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:42.212754 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e66929-08a3-4aa1-8a96-740f8f8eb7c8-metrics-certs\") pod \"network-metrics-daemon-skd9g\" (UID: \"78e66929-08a3-4aa1-8a96-740f8f8eb7c8\") " pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:18:42.426691 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:42.426664 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lwbh\"" Jul 31 14:18:42.435290 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:42.435266 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skd9g" Jul 31 14:18:42.577918 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:42.577888 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-skd9g"] Jul 31 14:18:42.580974 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:18:42.580949 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e66929_08a3_4aa1_8a96_740f8f8eb7c8.slice/crio-928a83e535a290d2055631a368de9659e2c16c5e0bbfeb2d9d27b551f9b52f57 WatchSource:0}: Error finding container 928a83e535a290d2055631a368de9659e2c16c5e0bbfeb2d9d27b551f9b52f57: Status 404 returned error can't find the container with id 928a83e535a290d2055631a368de9659e2c16c5e0bbfeb2d9d27b551f9b52f57 Jul 31 14:18:42.582878 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:42.582863 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 14:18:42.832058 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:42.831979 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-skd9g" event={"ID":"78e66929-08a3-4aa1-8a96-740f8f8eb7c8","Type":"ContainerStarted","Data":"928a83e535a290d2055631a368de9659e2c16c5e0bbfeb2d9d27b551f9b52f57"} Jul 31 14:18:44.842047 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:44.842016 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-skd9g" event={"ID":"78e66929-08a3-4aa1-8a96-740f8f8eb7c8","Type":"ContainerStarted","Data":"6e167fef868dd5051a2246485e8e40d14305fff592f70961ad13daf8d9d0fcc4"} Jul 31 14:18:44.842422 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:44.842063 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-skd9g" event={"ID":"78e66929-08a3-4aa1-8a96-740f8f8eb7c8","Type":"ContainerStarted","Data":"72f739ab125f7aa6d2e60cd1963a1b4c777ef8ecc6a9b141744e034dc1976de3"} Jul 31 14:18:44.855692 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:18:44.855649 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-skd9g" podStartSLOduration=375.530002422 podStartE2EDuration="6m16.855634093s" totalImagesPullingTime="1.325631671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:18:44.854680631 +0000 UTC m=+376.891503699" watchObservedRunningTime="2026-07-31 14:18:44.855634093 +0000 UTC m=+376.892457160" Jul 31 14:19:35.198884 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:19:35.198813 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:19:35.214037 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:19:35.214007 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:19:36.035376 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:19:36.035349 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 14:23:06.670822 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.670745 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22"] Jul 31 14:23:06.674044 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.674029 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" Jul 31 14:23:06.676093 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.676070 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 31 14:23:06.676698 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.676663 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 31 14:23:06.676698 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.676690 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 31 14:23:06.676904 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.676668 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4jnpn\"" Jul 31 14:23:06.676904 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.676745 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 31 14:23:06.676904 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.676752 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 31 14:23:06.683121 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.683103 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22"] Jul 31 14:23:06.763756 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.763727 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6a392d13-3092-4a3c-abd2-b84be867aa00-manager-config\") pod \"lws-controller-manager-58f6dff6d5-ckp22\" (UID: \"6a392d13-3092-4a3c-abd2-b84be867aa00\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" Jul 31 14:23:06.763925 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.763764 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a392d13-3092-4a3c-abd2-b84be867aa00-cert\") pod \"lws-controller-manager-58f6dff6d5-ckp22\" (UID: \"6a392d13-3092-4a3c-abd2-b84be867aa00\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" Jul 31 14:23:06.763925 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.763792 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6a392d13-3092-4a3c-abd2-b84be867aa00-metrics-cert\") pod \"lws-controller-manager-58f6dff6d5-ckp22\" (UID: \"6a392d13-3092-4a3c-abd2-b84be867aa00\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" Jul 31 14:23:06.763925 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.763890 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srxvp\" (UniqueName: \"kubernetes.io/projected/6a392d13-3092-4a3c-abd2-b84be867aa00-kube-api-access-srxvp\") pod \"lws-controller-manager-58f6dff6d5-ckp22\" (UID: \"6a392d13-3092-4a3c-abd2-b84be867aa00\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" Jul 31 14:23:06.865095 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.865068 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6a392d13-3092-4a3c-abd2-b84be867aa00-manager-config\") pod \"lws-controller-manager-58f6dff6d5-ckp22\" (UID: \"6a392d13-3092-4a3c-abd2-b84be867aa00\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" Jul 31 14:23:06.865186 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.865105 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a392d13-3092-4a3c-abd2-b84be867aa00-cert\") pod \"lws-controller-manager-58f6dff6d5-ckp22\" (UID: \"6a392d13-3092-4a3c-abd2-b84be867aa00\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" Jul 31 14:23:06.865186 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.865133 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6a392d13-3092-4a3c-abd2-b84be867aa00-metrics-cert\") pod \"lws-controller-manager-58f6dff6d5-ckp22\" (UID: \"6a392d13-3092-4a3c-abd2-b84be867aa00\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" Jul 31 14:23:06.865186 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.865170 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srxvp\" (UniqueName: \"kubernetes.io/projected/6a392d13-3092-4a3c-abd2-b84be867aa00-kube-api-access-srxvp\") pod \"lws-controller-manager-58f6dff6d5-ckp22\" (UID: \"6a392d13-3092-4a3c-abd2-b84be867aa00\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" Jul 31 14:23:06.865804 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.865782 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6a392d13-3092-4a3c-abd2-b84be867aa00-manager-config\") pod \"lws-controller-manager-58f6dff6d5-ckp22\" (UID: \"6a392d13-3092-4a3c-abd2-b84be867aa00\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" Jul 31 14:23:06.867449 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.867428 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a392d13-3092-4a3c-abd2-b84be867aa00-cert\") pod \"lws-controller-manager-58f6dff6d5-ckp22\" (UID: \"6a392d13-3092-4a3c-abd2-b84be867aa00\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" Jul 31 14:23:06.867604 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.867584 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6a392d13-3092-4a3c-abd2-b84be867aa00-metrics-cert\") pod \"lws-controller-manager-58f6dff6d5-ckp22\" (UID: \"6a392d13-3092-4a3c-abd2-b84be867aa00\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" Jul 31 14:23:06.877106 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.877088 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srxvp\" (UniqueName: \"kubernetes.io/projected/6a392d13-3092-4a3c-abd2-b84be867aa00-kube-api-access-srxvp\") pod \"lws-controller-manager-58f6dff6d5-ckp22\" (UID: \"6a392d13-3092-4a3c-abd2-b84be867aa00\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" Jul 31 14:23:06.983061 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:06.982977 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" Jul 31 14:23:07.106730 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:07.106701 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22"] Jul 31 14:23:07.109510 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:23:07.109479 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a392d13_3092_4a3c_abd2_b84be867aa00.slice/crio-37371f03ed397c79433c464fb81d189c166174e3f411e65e895c8d478eee5858 WatchSource:0}: Error finding container 37371f03ed397c79433c464fb81d189c166174e3f411e65e895c8d478eee5858: Status 404 returned error can't find the container with id 37371f03ed397c79433c464fb81d189c166174e3f411e65e895c8d478eee5858 Jul 31 14:23:07.759089 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:07.759052 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" event={"ID":"6a392d13-3092-4a3c-abd2-b84be867aa00","Type":"ContainerStarted","Data":"37371f03ed397c79433c464fb81d189c166174e3f411e65e895c8d478eee5858"} Jul 31 14:23:16.398755 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.398718 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4"] Jul 31 14:23:16.403183 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.403162 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:16.406680 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.406661 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 31 14:23:16.406897 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.406878 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 31 14:23:16.407102 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.407087 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 31 14:23:16.407601 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.407586 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 31 14:23:16.407662 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.407615 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2k6vh\"" Jul 31 14:23:16.419980 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.419958 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4"] Jul 31 14:23:16.541742 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.541705 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8-manifests\") pod \"opendatahub-operator-controller-manager-698c94c45b-pkqb4\" (UID: \"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:16.541935 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.541748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-698c94c45b-pkqb4\" (UID: \"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:16.541935 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.541782 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfv7h\" (UniqueName: \"kubernetes.io/projected/bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8-kube-api-access-qfv7h\") pod \"opendatahub-operator-controller-manager-698c94c45b-pkqb4\" (UID: \"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:16.541935 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.541816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8-webhook-cert\") pod \"opendatahub-operator-controller-manager-698c94c45b-pkqb4\" (UID: \"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:16.541935 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.541865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8-tmp\") pod \"opendatahub-operator-controller-manager-698c94c45b-pkqb4\" (UID: \"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:16.642246 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.642213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-698c94c45b-pkqb4\" (UID: \"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:16.642390 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.642261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfv7h\" (UniqueName: \"kubernetes.io/projected/bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8-kube-api-access-qfv7h\") pod \"opendatahub-operator-controller-manager-698c94c45b-pkqb4\" (UID: \"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:16.642390 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.642298 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8-webhook-cert\") pod \"opendatahub-operator-controller-manager-698c94c45b-pkqb4\" (UID: \"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:16.642390 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.642341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8-tmp\") pod \"opendatahub-operator-controller-manager-698c94c45b-pkqb4\" (UID: \"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:16.642390 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.642376 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8-manifests\") pod \"opendatahub-operator-controller-manager-698c94c45b-pkqb4\" (UID: \"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:16.642825 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.642800 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8-manifests\") pod \"opendatahub-operator-controller-manager-698c94c45b-pkqb4\" (UID: \"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:16.644573 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.644551 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8-tmp\") pod \"opendatahub-operator-controller-manager-698c94c45b-pkqb4\" (UID: \"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:16.644868 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.644832 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8-webhook-cert\") pod \"opendatahub-operator-controller-manager-698c94c45b-pkqb4\" (UID: \"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:16.644925 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.644905 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-698c94c45b-pkqb4\" (UID: \"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:16.654060 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.654002 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfv7h\" (UniqueName: \"kubernetes.io/projected/bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8-kube-api-access-qfv7h\") pod \"opendatahub-operator-controller-manager-698c94c45b-pkqb4\" (UID: \"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:16.713687 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.713656 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:16.842019 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:16.841988 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4"] Jul 31 14:23:16.845018 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:23:16.844989 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0d24d0_5cb8_4ee0_9e98_949dfabc8ec8.slice/crio-fba336e22c19b8d06b8ba039d6215d61b82cf4e6bec0448c307adee21c4f6e91 WatchSource:0}: Error finding container fba336e22c19b8d06b8ba039d6215d61b82cf4e6bec0448c307adee21c4f6e91: Status 404 returned error can't find the container with id fba336e22c19b8d06b8ba039d6215d61b82cf4e6bec0448c307adee21c4f6e91 Jul 31 14:23:17.795347 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:17.795312 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" event={"ID":"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8","Type":"ContainerStarted","Data":"fba336e22c19b8d06b8ba039d6215d61b82cf4e6bec0448c307adee21c4f6e91"} Jul 31 14:23:20.808674 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:20.808637 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" event={"ID":"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8","Type":"ContainerStarted","Data":"f90979d062382d3d6f592eebc5971eac2d78620f34a7a9f16639ad1b0bfaedf6"} Jul 31 14:23:21.813870 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:21.813819 2577 generic.go:356] "Generic (PLEG): container finished" podID="bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8" containerID="f90979d062382d3d6f592eebc5971eac2d78620f34a7a9f16639ad1b0bfaedf6" exitCode=0 Jul 31 14:23:21.814244 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:21.813906 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" event={"ID":"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8","Type":"ContainerDied","Data":"f90979d062382d3d6f592eebc5971eac2d78620f34a7a9f16639ad1b0bfaedf6"} Jul 31 14:23:22.819721 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:22.819683 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" event={"ID":"bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8","Type":"ContainerStarted","Data":"7020c0352177772ced07a1ff0ac4b226ea8c477621ce9a3b64be88b1aacb8951"} Jul 31 14:23:22.820119 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:22.819772 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:22.838722 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:22.838674 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" podStartSLOduration=2.584572296 podStartE2EDuration="6.838658186s" totalImagesPullingTime="3.232829424s" totalInitContainerRuntime="1.021256466s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:23:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:23:22.836376789 +0000 UTC m=+654.873199868" watchObservedRunningTime="2026-07-31 14:23:22.838658186 +0000 UTC m=+654.875481314" Jul 31 14:23:27.467320 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:23:27.467270 2577 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://registry.redhat.io/leader-worker-set/lws-rhel9@sha256:affb303b1173c273231bb50fef07310b0e220d2f08bfc0aa5912d0825e3e0d4f: reading manifest sha256:70099611ea7781bb7b113af98e8a85ddf0ab01d5fe804cf4fec44d66a569940f in registry.redhat.io/leader-worker-set/lws-rhel9: received unexpected HTTP status: 504 Gateway Time-out; artifact err: get manifest: build image source: pinging container registry registry.redhat.io: received unexpected HTTP status: 504 Gateway Time-out" image="registry.redhat.io/leader-worker-set/lws-rhel9@sha256:affb303b1173c273231bb50fef07310b0e220d2f08bfc0aa5912d0825e3e0d4f" Jul 31 14:23:27.467681 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:23:27.467401 2577 kuberuntime_manager.go:1459] "Unhandled Error" err="container manager start failed in pod lws-controller-manager-58f6dff6d5-ckp22_openshift-lws-operator(6a392d13-3092-4a3c-abd2-b84be867aa00): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://registry.redhat.io/leader-worker-set/lws-rhel9@sha256:affb303b1173c273231bb50fef07310b0e220d2f08bfc0aa5912d0825e3e0d4f: reading manifest sha256:70099611ea7781bb7b113af98e8a85ddf0ab01d5fe804cf4fec44d66a569940f in registry.redhat.io/leader-worker-set/lws-rhel9: received unexpected HTTP status: 504 Gateway Time-out; artifact err: get manifest: build image source: pinging container registry registry.redhat.io: received unexpected HTTP status: 504 Gateway Time-out" logger="UnhandledError" Jul 31 14:23:27.467681 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:23:27.467445 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://registry.redhat.io/leader-worker-set/lws-rhel9@sha256:affb303b1173c273231bb50fef07310b0e220d2f08bfc0aa5912d0825e3e0d4f: reading manifest sha256:70099611ea7781bb7b113af98e8a85ddf0ab01d5fe804cf4fec44d66a569940f in registry.redhat.io/leader-worker-set/lws-rhel9: received unexpected HTTP status: 504 Gateway Time-out; artifact err: get manifest: build image source: pinging container registry registry.redhat.io: received unexpected HTTP status: 504 Gateway Time-out\"" pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" podUID="6a392d13-3092-4a3c-abd2-b84be867aa00" Jul 31 14:23:27.838756 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:23:27.838706 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/leader-worker-set/lws-rhel9@sha256:affb303b1173c273231bb50fef07310b0e220d2f08bfc0aa5912d0825e3e0d4f\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://registry.redhat.io/leader-worker-set/lws-rhel9@sha256:affb303b1173c273231bb50fef07310b0e220d2f08bfc0aa5912d0825e3e0d4f: reading manifest sha256:70099611ea7781bb7b113af98e8a85ddf0ab01d5fe804cf4fec44d66a569940f in registry.redhat.io/leader-worker-set/lws-rhel9: received unexpected HTTP status: 504 Gateway Time-out; artifact err: get manifest: build image source: pinging container registry registry.redhat.io: received unexpected HTTP status: 504 Gateway Time-out\"" pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" podUID="6a392d13-3092-4a3c-abd2-b84be867aa00" Jul 31 14:23:33.825938 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:33.825903 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-pkqb4" Jul 31 14:23:42.895520 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:42.895485 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" event={"ID":"6a392d13-3092-4a3c-abd2-b84be867aa00","Type":"ContainerStarted","Data":"0dcb738acac72e3424da9f5f2ea92abd9c18bd2cf57415745dfb0cd1805b75d5"} Jul 31 14:23:42.896027 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:42.895696 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" Jul 31 14:23:42.923197 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:42.923148 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" podStartSLOduration=2.094739339 podStartE2EDuration="36.923134443s" totalImagesPullingTime="34.828395104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-31 14:23:42.92241671 +0000 UTC m=+674.959239779" watchObservedRunningTime="2026-07-31 14:23:42.923134443 +0000 UTC m=+674.959957509" Jul 31 14:23:43.697516 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.697481 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4"] Jul 31 14:23:43.700635 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.700618 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.703737 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.703718 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 31 14:23:43.704872 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.704856 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 31 14:23:43.704872 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.704864 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 31 14:23:43.705166 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.705144 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 31 14:23:43.705240 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.705166 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 31 14:23:43.705240 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.705183 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-l2qsb\"" Jul 31 14:23:43.705240 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.705196 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 31 14:23:43.740105 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.740078 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4"] Jul 31 14:23:43.754145 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.754120 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/431d2c6a-3bd3-4d06-af77-417f21012958-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.754243 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.754148 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/431d2c6a-3bd3-4d06-af77-417f21012958-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.754243 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.754185 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/431d2c6a-3bd3-4d06-af77-417f21012958-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.754356 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.754243 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/431d2c6a-3bd3-4d06-af77-417f21012958-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.754356 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.754284 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mgwv\" (UniqueName: \"kubernetes.io/projected/431d2c6a-3bd3-4d06-af77-417f21012958-kube-api-access-7mgwv\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.754356 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.754332 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/431d2c6a-3bd3-4d06-af77-417f21012958-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.754448 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.754384 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/431d2c6a-3bd3-4d06-af77-417f21012958-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.855349 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.855323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/431d2c6a-3bd3-4d06-af77-417f21012958-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.855473 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.855366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/431d2c6a-3bd3-4d06-af77-417f21012958-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.855473 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.855387 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/431d2c6a-3bd3-4d06-af77-417f21012958-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.855473 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.855424 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/431d2c6a-3bd3-4d06-af77-417f21012958-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.855473 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.855440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/431d2c6a-3bd3-4d06-af77-417f21012958-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.855473 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.855465 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mgwv\" (UniqueName: \"kubernetes.io/projected/431d2c6a-3bd3-4d06-af77-417f21012958-kube-api-access-7mgwv\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.855724 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.855602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/431d2c6a-3bd3-4d06-af77-417f21012958-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.856132 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.856107 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/431d2c6a-3bd3-4d06-af77-417f21012958-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.857815 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.857792 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/431d2c6a-3bd3-4d06-af77-417f21012958-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.857815 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.857805 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/431d2c6a-3bd3-4d06-af77-417f21012958-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.857970 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.857793 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/431d2c6a-3bd3-4d06-af77-417f21012958-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.857970 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.857858 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/431d2c6a-3bd3-4d06-af77-417f21012958-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.864945 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.864915 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/431d2c6a-3bd3-4d06-af77-417f21012958-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:43.865137 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:43.865115 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mgwv\" (UniqueName: \"kubernetes.io/projected/431d2c6a-3bd3-4d06-af77-417f21012958-kube-api-access-7mgwv\") pod \"istiod-openshift-gateway-5677ccb49b-6k8v4\" (UID: \"431d2c6a-3bd3-4d06-af77-417f21012958\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:44.010007 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:44.009924 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:44.153423 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:44.153394 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4"] Jul 31 14:23:44.156524 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:23:44.156495 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431d2c6a_3bd3_4d06_af77_417f21012958.slice/crio-3b2ad9fe81200d8cdb81cf25a5bab2573908463ddc388b649d22a13cfd7fe321 WatchSource:0}: Error finding container 3b2ad9fe81200d8cdb81cf25a5bab2573908463ddc388b649d22a13cfd7fe321: Status 404 returned error can't find the container with id 3b2ad9fe81200d8cdb81cf25a5bab2573908463ddc388b649d22a13cfd7fe321 Jul 31 14:23:44.158954 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:44.158936 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 14:23:44.904153 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:44.904117 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" event={"ID":"431d2c6a-3bd3-4d06-af77-417f21012958","Type":"ContainerStarted","Data":"3b2ad9fe81200d8cdb81cf25a5bab2573908463ddc388b649d22a13cfd7fe321"} Jul 31 14:23:48.887789 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:48.887750 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:23:48.888080 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:48.887826 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:23:49.925545 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:49.925514 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" event={"ID":"431d2c6a-3bd3-4d06-af77-417f21012958","Type":"ContainerStarted","Data":"ff8ef8610757717b0934607271cc061f8068e3930d5a163ec70f7f3198f2e833"} Jul 31 14:23:49.926001 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:49.925668 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:49.927317 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:49.927298 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" Jul 31 14:23:49.946607 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:49.946560 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6k8v4" podStartSLOduration=2.218115782 podStartE2EDuration="6.946544292s" totalImagesPullingTime="4.72842851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:23:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:23:49.945707724 +0000 UTC m=+681.982530792" watchObservedRunningTime="2026-07-31 14:23:49.946544292 +0000 UTC m=+681.983367361" Jul 31 14:23:53.901832 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:23:53.901800 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-ckp22" Jul 31 14:26:00.470588 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:00.470511 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:26:00.473589 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:00.473574 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx" Jul 31 14:26:00.475247 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:00.475220 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 31 14:26:00.475371 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:00.475280 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gx8vx\"" Jul 31 14:26:00.475754 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:00.475739 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 31 14:26:00.475803 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:00.475766 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 31 14:26:00.481659 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:00.481635 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:26:00.510264 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:00.510231 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:26:00.584302 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:00.584269 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk4p9\" (UniqueName: \"kubernetes.io/projected/021631ac-a079-4b7d-bf1d-42bc81796fc0-kube-api-access-rk4p9\") pod \"limitador-limitador-85d9db4d5d-bqzgx\" (UID: \"021631ac-a079-4b7d-bf1d-42bc81796fc0\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx" Jul 31 14:26:00.584475 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:00.584326 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/021631ac-a079-4b7d-bf1d-42bc81796fc0-config-file\") pod \"limitador-limitador-85d9db4d5d-bqzgx\" (UID: \"021631ac-a079-4b7d-bf1d-42bc81796fc0\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx" Jul 31 14:26:00.685567 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:00.685534 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk4p9\" (UniqueName: \"kubernetes.io/projected/021631ac-a079-4b7d-bf1d-42bc81796fc0-kube-api-access-rk4p9\") pod \"limitador-limitador-85d9db4d5d-bqzgx\" (UID: \"021631ac-a079-4b7d-bf1d-42bc81796fc0\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx" Jul 31 14:26:00.685683 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:00.685583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/021631ac-a079-4b7d-bf1d-42bc81796fc0-config-file\") pod \"limitador-limitador-85d9db4d5d-bqzgx\" (UID: \"021631ac-a079-4b7d-bf1d-42bc81796fc0\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx" Jul 31 14:26:00.686237 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:00.686219 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/021631ac-a079-4b7d-bf1d-42bc81796fc0-config-file\") pod \"limitador-limitador-85d9db4d5d-bqzgx\" (UID: \"021631ac-a079-4b7d-bf1d-42bc81796fc0\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx" Jul 31 14:26:00.693047 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:00.693026 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk4p9\" (UniqueName: \"kubernetes.io/projected/021631ac-a079-4b7d-bf1d-42bc81796fc0-kube-api-access-rk4p9\") pod \"limitador-limitador-85d9db4d5d-bqzgx\" (UID: \"021631ac-a079-4b7d-bf1d-42bc81796fc0\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx" Jul 31 14:26:00.784064 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:00.784001 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx" Jul 31 14:26:00.909342 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:00.909310 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:26:00.912349 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:26:00.912309 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021631ac_a079_4b7d_bf1d_42bc81796fc0.slice/crio-6cfd079f6fbaadd8d1b90e0c5dc1b46b0904018409660e8b30c58dc198ddda5c WatchSource:0}: Error finding container 6cfd079f6fbaadd8d1b90e0c5dc1b46b0904018409660e8b30c58dc198ddda5c: Status 404 returned error can't find the container with id 6cfd079f6fbaadd8d1b90e0c5dc1b46b0904018409660e8b30c58dc198ddda5c Jul 31 14:26:01.410065 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:01.410032 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx" event={"ID":"021631ac-a079-4b7d-bf1d-42bc81796fc0","Type":"ContainerStarted","Data":"6cfd079f6fbaadd8d1b90e0c5dc1b46b0904018409660e8b30c58dc198ddda5c"} Jul 31 14:26:04.423765 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:04.423723 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx" event={"ID":"021631ac-a079-4b7d-bf1d-42bc81796fc0","Type":"ContainerStarted","Data":"86b19cbf81578e6bb5b6378a9bbf0c55550ed1efa63e0e98b42822c8cac99c0e"} Jul 31 14:26:04.428107 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:04.428080 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx" Jul 31 14:26:04.442561 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:04.442512 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx" podStartSLOduration=1.839382374 podStartE2EDuration="4.44249702s" totalImagesPullingTime="2.603114646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:26:04.441337703 +0000 UTC m=+816.478160770" watchObservedRunningTime="2026-07-31 14:26:04.44249702 +0000 UTC m=+816.479320087" Jul 31 14:26:16.432156 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:26:16.432123 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx" Jul 31 14:28:17.450159 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:17.450120 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:28:43.083223 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:43.083183 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:28:49.871319 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:49.871282 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f"] Jul 31 14:28:49.874600 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:49.874584 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:49.877078 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:49.877053 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 31 14:28:49.877078 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:49.877052 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 31 14:28:49.877265 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:49.877057 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7xdgk\"" Jul 31 14:28:49.877265 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:49.877119 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 31 14:28:49.884832 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:49.884810 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f"] Jul 31 14:28:49.931414 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:49.931385 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3880ffab-f747-47fb-bfac-d7eb278eb474-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xkd9f\" (UID: \"3880ffab-f747-47fb-bfac-d7eb278eb474\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:49.931589 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:49.931418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3880ffab-f747-47fb-bfac-d7eb278eb474-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xkd9f\" (UID: \"3880ffab-f747-47fb-bfac-d7eb278eb474\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:49.931589 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:49.931463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3880ffab-f747-47fb-bfac-d7eb278eb474-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xkd9f\" (UID: \"3880ffab-f747-47fb-bfac-d7eb278eb474\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:49.931589 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:49.931494 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3880ffab-f747-47fb-bfac-d7eb278eb474-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xkd9f\" (UID: \"3880ffab-f747-47fb-bfac-d7eb278eb474\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:49.931589 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:49.931569 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dhnf\" (UniqueName: \"kubernetes.io/projected/3880ffab-f747-47fb-bfac-d7eb278eb474-kube-api-access-6dhnf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xkd9f\" (UID: \"3880ffab-f747-47fb-bfac-d7eb278eb474\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:49.931752 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:49.931595 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3880ffab-f747-47fb-bfac-d7eb278eb474-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xkd9f\" (UID: \"3880ffab-f747-47fb-bfac-d7eb278eb474\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:50.032673 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:50.032637 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3880ffab-f747-47fb-bfac-d7eb278eb474-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xkd9f\" (UID: \"3880ffab-f747-47fb-bfac-d7eb278eb474\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:50.032673 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:50.032679 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3880ffab-f747-47fb-bfac-d7eb278eb474-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xkd9f\" (UID: \"3880ffab-f747-47fb-bfac-d7eb278eb474\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:50.032969 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:50.032725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dhnf\" (UniqueName: \"kubernetes.io/projected/3880ffab-f747-47fb-bfac-d7eb278eb474-kube-api-access-6dhnf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xkd9f\" (UID: \"3880ffab-f747-47fb-bfac-d7eb278eb474\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:50.032969 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:50.032744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3880ffab-f747-47fb-bfac-d7eb278eb474-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xkd9f\" (UID: \"3880ffab-f747-47fb-bfac-d7eb278eb474\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:50.032969 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:50.032777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3880ffab-f747-47fb-bfac-d7eb278eb474-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xkd9f\" (UID: \"3880ffab-f747-47fb-bfac-d7eb278eb474\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:50.032969 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:50.032795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3880ffab-f747-47fb-bfac-d7eb278eb474-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xkd9f\" (UID: \"3880ffab-f747-47fb-bfac-d7eb278eb474\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:50.033265 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:50.033239 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3880ffab-f747-47fb-bfac-d7eb278eb474-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xkd9f\" (UID: \"3880ffab-f747-47fb-bfac-d7eb278eb474\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:50.033388 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:50.033276 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3880ffab-f747-47fb-bfac-d7eb278eb474-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xkd9f\" (UID: \"3880ffab-f747-47fb-bfac-d7eb278eb474\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:50.033516 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:50.033491 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3880ffab-f747-47fb-bfac-d7eb278eb474-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xkd9f\" (UID: \"3880ffab-f747-47fb-bfac-d7eb278eb474\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:50.035178 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:50.035156 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3880ffab-f747-47fb-bfac-d7eb278eb474-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xkd9f\" (UID: \"3880ffab-f747-47fb-bfac-d7eb278eb474\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:50.035382 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:50.035366 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3880ffab-f747-47fb-bfac-d7eb278eb474-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xkd9f\" (UID: \"3880ffab-f747-47fb-bfac-d7eb278eb474\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:50.039498 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:50.039473 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dhnf\" (UniqueName: \"kubernetes.io/projected/3880ffab-f747-47fb-bfac-d7eb278eb474-kube-api-access-6dhnf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xkd9f\" (UID: \"3880ffab-f747-47fb-bfac-d7eb278eb474\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:50.184400 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:50.184359 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:28:50.312986 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:50.312941 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f"] Jul 31 14:28:50.315758 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:28:50.315731 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3880ffab_f747_47fb_bfac_d7eb278eb474.slice/crio-a2e5273f5c27a5ba05722d7818b5a2fa7c7ba80c9075b51221e13163a4e17e29 WatchSource:0}: Error finding container a2e5273f5c27a5ba05722d7818b5a2fa7c7ba80c9075b51221e13163a4e17e29: Status 404 returned error can't find the container with id a2e5273f5c27a5ba05722d7818b5a2fa7c7ba80c9075b51221e13163a4e17e29 Jul 31 14:28:50.317012 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:50.316990 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 14:28:50.779596 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:50.779562 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:28:51.042075 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:51.041981 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" event={"ID":"3880ffab-f747-47fb-bfac-d7eb278eb474","Type":"ContainerStarted","Data":"a2e5273f5c27a5ba05722d7818b5a2fa7c7ba80c9075b51221e13163a4e17e29"} Jul 31 14:28:52.047589 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:52.047556 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" event={"ID":"3880ffab-f747-47fb-bfac-d7eb278eb474","Type":"ContainerStarted","Data":"49f19877dc2e98f22333ed860da4dff2745056cfd3cff962f458da7aa3c5a5e4"} Jul 31 14:28:52.065662 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:28:52.065606 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" podStartSLOduration=2.051216836 podStartE2EDuration="3.065582569s" totalImagesPullingTime="1.014365733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:28:52.065364233 +0000 UTC m=+984.102187325" watchObservedRunningTime="2026-07-31 14:28:52.065582569 +0000 UTC m=+984.102405637" Jul 31 14:29:00.184767 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:29:00.184727 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:29:00.185282 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:29:00.184799 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:29:00.198352 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:29:00.198329 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:29:01.094236 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:29:01.094206 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xkd9f" Jul 31 14:29:05.373625 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:29:05.373590 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:29:16.496895 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:29:16.496859 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:29:23.378862 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:29:23.378817 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:29:34.882660 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:29:34.882624 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:31:03.273407 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:31:03.273368 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:31:12.472029 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:31:12.471996 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:31:24.271440 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:31:24.271400 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:33:05.071824 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:33:05.071791 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:33:15.573499 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:33:15.573467 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:34:19.474438 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:34:19.474402 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:34:26.476197 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:34:26.476150 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:34:42.077547 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:34:42.077500 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:34:48.078513 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:34:48.078476 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:34:57.574383 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:34:57.574295 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:35:03.969986 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:35:03.969954 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:35:18.376954 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:35:18.376904 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:35:23.776484 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:35:23.776448 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:35:31.775131 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:35:31.775089 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:35:40.775135 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:35:40.775106 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:35:53.476426 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:35:53.476387 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:36:04.083720 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:36:04.083679 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:36:26.870267 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:36:26.870179 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:36:37.368224 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:36:37.368185 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:36:43.174359 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:36:43.174325 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:36:50.674422 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:36:50.674389 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:36:58.179534 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:36:58.179487 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:37:03.077420 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:37:03.077382 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:37:09.081443 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:37:09.081410 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:37:16.577197 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:37:16.577160 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:37:28.273666 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:37:28.273634 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:37:35.267397 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:37:35.267367 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:37:46.475226 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:37:46.475189 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:37:51.774246 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:37:51.774166 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:37:58.175070 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:37:58.175037 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:38:06.867024 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:38:06.866985 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:38:12.076509 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:38:12.076474 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:38:22.872655 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:38:22.872620 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:40:39.868893 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:40:39.868855 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:40:44.974828 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:40:44.974795 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:41:09.782993 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:41:09.782909 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:42:12.874972 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:42:12.874934 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:42:23.170204 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:42:23.170125 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:42:27.376342 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:42:27.376310 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:42:32.380984 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:42:32.380952 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:42:42.677603 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:42:42.677561 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:42:47.575982 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:42:47.575944 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:42:57.777761 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:42:57.777727 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:43:04.484394 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:43:04.484361 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:43:13.173970 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:43:13.173935 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:43:19.578576 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:43:19.578541 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:43:29.082902 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:43:29.082866 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:43:51.276758 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:43:51.276677 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:43:57.780681 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:43:57.780644 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:44:17.777116 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:44:17.777078 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:44:50.178122 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:44:50.178090 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:45:16.874858 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:45:16.874814 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:45:26.871595 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:45:26.871558 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:45:32.571337 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:45:32.571304 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:45:41.486989 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:45:41.486954 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:45:48.976349 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:45:48.976317 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:45:55.267780 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:45:55.267744 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:46:03.073712 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:46:03.073671 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:46:11.380471 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:46:11.380432 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:46:21.572034 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:46:21.571998 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:46:28.077508 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:46:28.077464 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:46:44.273935 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:46:44.273894 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:46:50.873275 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:46:50.873195 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:47:02.672967 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:47:02.672932 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:47:09.371371 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:47:09.371338 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:47:16.971812 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:47:16.971780 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:47:22.077470 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:47:22.077426 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:47:29.780051 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:47:29.780018 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:47:35.768562 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:47:35.768523 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:47:41.977046 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:47:41.977011 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:47:52.675574 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:47:52.675542 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:47:58.872720 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:47:58.872685 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:48:09.879304 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:48:09.879264 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:48:25.170924 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:48:25.170830 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:48:31.178972 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:48:31.178927 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:48:37.569056 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:48:37.569019 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:48:49.371292 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:48:49.371252 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:48:57.579667 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:48:57.579628 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:49:03.079858 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:03.079806 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:49:15.295437 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.295403 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md"] Jul 31 14:49:15.298555 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.298540 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md" Jul 31 14:49:15.300603 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.300582 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3ea37d42\"" Jul 31 14:49:15.300704 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.300619 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3ea37d42-dockercfg-qfdpf\"" Jul 31 14:49:15.306805 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.306779 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md"] Jul 31 14:49:15.324814 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.324789 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c"] Jul 31 14:49:15.327886 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.327870 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c" Jul 31 14:49:15.337782 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.337764 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c"] Jul 31 14:49:15.376089 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.376062 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mhjr\" (UniqueName: \"kubernetes.io/projected/cdc7b02e-48b0-4915-b6c5-f5515237297f-kube-api-access-9mhjr\") pod \"payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md\" (UID: \"cdc7b02e-48b0-4915-b6c5-f5515237297f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md" Jul 31 14:49:15.376213 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.376196 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc7b02e-48b0-4915-b6c5-f5515237297f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md\" (UID: \"cdc7b02e-48b0-4915-b6c5-f5515237297f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md" Jul 31 14:49:15.477031 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.477000 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfpg9\" (UniqueName: \"kubernetes.io/projected/1244ce10-52bd-4990-95cf-b99da4513a36-kube-api-access-sfpg9\") pod \"payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c\" (UID: \"1244ce10-52bd-4990-95cf-b99da4513a36\") " pod="openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c" Jul 31 14:49:15.477191 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.477046 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1244ce10-52bd-4990-95cf-b99da4513a36-plugins-config-volume\") pod \"payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c\" (UID: \"1244ce10-52bd-4990-95cf-b99da4513a36\") " pod="openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c" Jul 31 14:49:15.477191 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.477123 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc7b02e-48b0-4915-b6c5-f5515237297f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md\" (UID: \"cdc7b02e-48b0-4915-b6c5-f5515237297f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md" Jul 31 14:49:15.477191 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.477152 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mhjr\" (UniqueName: \"kubernetes.io/projected/cdc7b02e-48b0-4915-b6c5-f5515237297f-kube-api-access-9mhjr\") pod \"payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md\" (UID: \"cdc7b02e-48b0-4915-b6c5-f5515237297f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md" Jul 31 14:49:15.477888 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.477870 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc7b02e-48b0-4915-b6c5-f5515237297f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md\" (UID: \"cdc7b02e-48b0-4915-b6c5-f5515237297f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md" Jul 31 14:49:15.484101 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.484081 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mhjr\" (UniqueName: \"kubernetes.io/projected/cdc7b02e-48b0-4915-b6c5-f5515237297f-kube-api-access-9mhjr\") pod \"payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md\" (UID: \"cdc7b02e-48b0-4915-b6c5-f5515237297f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md" Jul 31 14:49:15.578267 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.578203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfpg9\" (UniqueName: \"kubernetes.io/projected/1244ce10-52bd-4990-95cf-b99da4513a36-kube-api-access-sfpg9\") pod \"payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c\" (UID: \"1244ce10-52bd-4990-95cf-b99da4513a36\") " pod="openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c" Jul 31 14:49:15.578267 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.578242 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1244ce10-52bd-4990-95cf-b99da4513a36-plugins-config-volume\") pod \"payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c\" (UID: \"1244ce10-52bd-4990-95cf-b99da4513a36\") " pod="openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c" Jul 31 14:49:15.578837 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.578820 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1244ce10-52bd-4990-95cf-b99da4513a36-plugins-config-volume\") pod \"payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c\" (UID: \"1244ce10-52bd-4990-95cf-b99da4513a36\") " pod="openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c" Jul 31 14:49:15.585395 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.585377 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfpg9\" (UniqueName: \"kubernetes.io/projected/1244ce10-52bd-4990-95cf-b99da4513a36-kube-api-access-sfpg9\") pod \"payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c\" (UID: \"1244ce10-52bd-4990-95cf-b99da4513a36\") " pod="openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c" Jul 31 14:49:15.609518 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.609490 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md" Jul 31 14:49:15.637189 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.637168 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c" Jul 31 14:49:15.745553 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.745527 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md"] Jul 31 14:49:15.749564 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:49:15.749531 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc7b02e_48b0_4915_b6c5_f5515237297f.slice/crio-4eec9c24bf914d6fe2a3d01c428d2a5a0821c52e8af35e87dcef588f99870948 WatchSource:0}: Error finding container 4eec9c24bf914d6fe2a3d01c428d2a5a0821c52e8af35e87dcef588f99870948: Status 404 returned error can't find the container with id 4eec9c24bf914d6fe2a3d01c428d2a5a0821c52e8af35e87dcef588f99870948 Jul 31 14:49:15.751393 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.751374 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 14:49:15.775628 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.775606 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c"] Jul 31 14:49:15.777986 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:49:15.777959 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1244ce10_52bd_4990_95cf_b99da4513a36.slice/crio-ce089d5ef3c60d02fae82e8c5a61e08077f5b6eeb605546ce5dac3cea2aa38c4 WatchSource:0}: Error finding container ce089d5ef3c60d02fae82e8c5a61e08077f5b6eeb605546ce5dac3cea2aa38c4: Status 404 returned error can't find the container with id ce089d5ef3c60d02fae82e8c5a61e08077f5b6eeb605546ce5dac3cea2aa38c4 Jul 31 14:49:15.954028 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.953989 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md" event={"ID":"cdc7b02e-48b0-4915-b6c5-f5515237297f","Type":"ContainerStarted","Data":"4eec9c24bf914d6fe2a3d01c428d2a5a0821c52e8af35e87dcef588f99870948"} Jul 31 14:49:15.955009 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:15.954982 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c" event={"ID":"1244ce10-52bd-4990-95cf-b99da4513a36","Type":"ContainerStarted","Data":"ce089d5ef3c60d02fae82e8c5a61e08077f5b6eeb605546ce5dac3cea2aa38c4"} Jul 31 14:49:17.368180 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:17.368136 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:49:18.971834 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:18.971790 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md" event={"ID":"cdc7b02e-48b0-4915-b6c5-f5515237297f","Type":"ContainerStarted","Data":"e17a8b7c2f51bca1265a6eeac4702880f1a752706e7a745868ab11f1c173ebb6"} Jul 31 14:49:18.972309 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:18.971885 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md" Jul 31 14:49:18.973152 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:18.973130 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c" event={"ID":"1244ce10-52bd-4990-95cf-b99da4513a36","Type":"ContainerStarted","Data":"1f829aa64cd75ede4db1ce30bd6d7f8b96607983e2bbc25a56644ecf0ecb4b62"} Jul 31 14:49:18.973227 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:18.973178 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c" Jul 31 14:49:18.985934 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:18.985880 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md" podStartSLOduration=1.483837649 podStartE2EDuration="3.985863285s" totalImagesPullingTime="2.502025636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:18.984975816 +0000 UTC m=+2211.021798882" watchObservedRunningTime="2026-07-31 14:49:18.985863285 +0000 UTC m=+2211.022686347" Jul 31 14:49:18.998977 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:18.998936 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c" podStartSLOduration=1.519618269 podStartE2EDuration="3.998921646s" totalImagesPullingTime="2.479303377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:18.997769911 +0000 UTC m=+2211.034592993" watchObservedRunningTime="2026-07-31 14:49:18.998921646 +0000 UTC m=+2211.035744713" Jul 31 14:49:29.980038 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:29.980002 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md" Jul 31 14:49:29.980446 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:29.980087 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c" Jul 31 14:49:39.220785 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:39.220749 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ea37d42-f979k"] Jul 31 14:49:39.224174 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:39.224125 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ea37d42-f979k" Jul 31 14:49:39.226008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:39.225981 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 14:49:39.226104 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:39.226031 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\"" Jul 31 14:49:39.231259 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:39.231221 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ea37d42-f979k"] Jul 31 14:49:39.384165 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:39.384128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ec8184-dbfc-4635-a7e6-4769def50dda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ea37d42-f979k\" (UID: \"f4ec8184-dbfc-4635-a7e6-4769def50dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ea37d42-f979k" Jul 31 14:49:39.485520 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:39.485448 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ec8184-dbfc-4635-a7e6-4769def50dda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ea37d42-f979k\" (UID: \"f4ec8184-dbfc-4635-a7e6-4769def50dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ea37d42-f979k" Jul 31 14:49:39.486050 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:39.486032 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ec8184-dbfc-4635-a7e6-4769def50dda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ea37d42-f979k\" (UID: \"f4ec8184-dbfc-4635-a7e6-4769def50dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ea37d42-f979k" Jul 31 14:49:39.535559 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:39.535536 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ea37d42-f979k" Jul 31 14:49:39.657990 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:39.657966 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ea37d42-f979k"] Jul 31 14:49:39.660323 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:49:39.660298 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ec8184_dbfc_4635_a7e6_4769def50dda.slice/crio-86950d772b5cf55d73b2ac7382c8020eef17772270b48046e2c3d00cb006117a WatchSource:0}: Error finding container 86950d772b5cf55d73b2ac7382c8020eef17772270b48046e2c3d00cb006117a: Status 404 returned error can't find the container with id 86950d772b5cf55d73b2ac7382c8020eef17772270b48046e2c3d00cb006117a Jul 31 14:49:40.061796 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:40.061764 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ea37d42-f979k" event={"ID":"f4ec8184-dbfc-4635-a7e6-4769def50dda","Type":"ContainerStarted","Data":"86950d772b5cf55d73b2ac7382c8020eef17772270b48046e2c3d00cb006117a"} Jul 31 14:49:42.071669 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:42.071538 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ea37d42-f979k" event={"ID":"f4ec8184-dbfc-4635-a7e6-4769def50dda","Type":"ContainerStarted","Data":"bcc84c26d8b95cd8fda8f51d3853ccb883f03dfa353bedb87cee895bd7bca1da"} Jul 31 14:49:42.084778 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:42.084733 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ea37d42-f979k" podStartSLOduration=1.00088739 podStartE2EDuration="3.084717715s" totalImagesPullingTime="2.083830325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:42.083878193 +0000 UTC m=+2234.120701259" watchObservedRunningTime="2026-07-31 14:49:42.084717715 +0000 UTC m=+2234.121540782" Jul 31 14:49:43.076974 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:43.076937 2577 generic.go:356] "Generic (PLEG): container finished" podID="f4ec8184-dbfc-4635-a7e6-4769def50dda" containerID="bcc84c26d8b95cd8fda8f51d3853ccb883f03dfa353bedb87cee895bd7bca1da" exitCode=0 Jul 31 14:49:43.077386 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:43.077025 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ea37d42-f979k" event={"ID":"f4ec8184-dbfc-4635-a7e6-4769def50dda","Type":"ContainerDied","Data":"bcc84c26d8b95cd8fda8f51d3853ccb883f03dfa353bedb87cee895bd7bca1da"} Jul 31 14:49:44.209768 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:44.209740 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ea37d42-f979k" Jul 31 14:49:44.222916 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:44.222893 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ec8184-dbfc-4635-a7e6-4769def50dda-maas-api-service-ca\") pod \"f4ec8184-dbfc-4635-a7e6-4769def50dda\" (UID: \"f4ec8184-dbfc-4635-a7e6-4769def50dda\") " Jul 31 14:49:44.223304 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:44.223274 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ec8184-dbfc-4635-a7e6-4769def50dda-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f4ec8184-dbfc-4635-a7e6-4769def50dda" (UID: "f4ec8184-dbfc-4635-a7e6-4769def50dda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:49:44.324391 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:44.324361 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ec8184-dbfc-4635-a7e6-4769def50dda-maas-api-service-ca\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:49:45.087239 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:45.087205 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ea37d42-f979k" event={"ID":"f4ec8184-dbfc-4635-a7e6-4769def50dda","Type":"ContainerDied","Data":"86950d772b5cf55d73b2ac7382c8020eef17772270b48046e2c3d00cb006117a"} Jul 31 14:49:45.087239 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:45.087228 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ea37d42-f979k" Jul 31 14:49:45.087512 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:45.087255 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86950d772b5cf55d73b2ac7382c8020eef17772270b48046e2c3d00cb006117a" Jul 31 14:49:55.213685 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.213595 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c"] Jul 31 14:49:55.214176 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.213917 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c" podUID="1244ce10-52bd-4990-95cf-b99da4513a36" containerName="payload-processing" containerID="cri-o://1f829aa64cd75ede4db1ce30bd6d7f8b96607983e2bbc25a56644ecf0ecb4b62" gracePeriod=30 Jul 31 14:49:55.219561 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.219534 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md"] Jul 31 14:49:55.219975 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.219920 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md" podUID="cdc7b02e-48b0-4915-b6c5-f5515237297f" containerName="payload-pre-processing" containerID="cri-o://e17a8b7c2f51bca1265a6eeac4702880f1a752706e7a745868ab11f1c173ebb6" gracePeriod=30 Jul 31 14:49:55.339635 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.339606 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ea37d42-f979k"] Jul 31 14:49:55.342887 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.342861 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ea37d42-f979k"] Jul 31 14:49:55.496910 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.496740 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c" Jul 31 14:49:55.504257 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.504234 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md" Jul 31 14:49:55.612589 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.612559 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mhjr\" (UniqueName: \"kubernetes.io/projected/cdc7b02e-48b0-4915-b6c5-f5515237297f-kube-api-access-9mhjr\") pod \"cdc7b02e-48b0-4915-b6c5-f5515237297f\" (UID: \"cdc7b02e-48b0-4915-b6c5-f5515237297f\") " Jul 31 14:49:55.612780 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.612609 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1244ce10-52bd-4990-95cf-b99da4513a36-plugins-config-volume\") pod \"1244ce10-52bd-4990-95cf-b99da4513a36\" (UID: \"1244ce10-52bd-4990-95cf-b99da4513a36\") " Jul 31 14:49:55.612780 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.612692 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfpg9\" (UniqueName: \"kubernetes.io/projected/1244ce10-52bd-4990-95cf-b99da4513a36-kube-api-access-sfpg9\") pod \"1244ce10-52bd-4990-95cf-b99da4513a36\" (UID: \"1244ce10-52bd-4990-95cf-b99da4513a36\") " Jul 31 14:49:55.612780 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.612725 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc7b02e-48b0-4915-b6c5-f5515237297f-plugins-config-volume\") pod \"cdc7b02e-48b0-4915-b6c5-f5515237297f\" (UID: \"cdc7b02e-48b0-4915-b6c5-f5515237297f\") " Jul 31 14:49:55.613131 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.613092 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1244ce10-52bd-4990-95cf-b99da4513a36-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1244ce10-52bd-4990-95cf-b99da4513a36" (UID: "1244ce10-52bd-4990-95cf-b99da4513a36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:49:55.613240 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.613214 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc7b02e-48b0-4915-b6c5-f5515237297f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cdc7b02e-48b0-4915-b6c5-f5515237297f" (UID: "cdc7b02e-48b0-4915-b6c5-f5515237297f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:49:55.614908 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.614884 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1244ce10-52bd-4990-95cf-b99da4513a36-kube-api-access-sfpg9" (OuterVolumeSpecName: "kube-api-access-sfpg9") pod "1244ce10-52bd-4990-95cf-b99da4513a36" (UID: "1244ce10-52bd-4990-95cf-b99da4513a36"). InnerVolumeSpecName "kube-api-access-sfpg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:49:55.614908 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.614896 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc7b02e-48b0-4915-b6c5-f5515237297f-kube-api-access-9mhjr" (OuterVolumeSpecName: "kube-api-access-9mhjr") pod "cdc7b02e-48b0-4915-b6c5-f5515237297f" (UID: "cdc7b02e-48b0-4915-b6c5-f5515237297f"). InnerVolumeSpecName "kube-api-access-9mhjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:49:55.713977 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.713929 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc7b02e-48b0-4915-b6c5-f5515237297f-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:49:55.713977 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.713972 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mhjr\" (UniqueName: \"kubernetes.io/projected/cdc7b02e-48b0-4915-b6c5-f5515237297f-kube-api-access-9mhjr\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:49:55.714118 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.713986 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1244ce10-52bd-4990-95cf-b99da4513a36-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:49:55.714118 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:55.713998 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfpg9\" (UniqueName: \"kubernetes.io/projected/1244ce10-52bd-4990-95cf-b99da4513a36-kube-api-access-sfpg9\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:49:56.134079 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.134041 2577 generic.go:356] "Generic (PLEG): container finished" podID="cdc7b02e-48b0-4915-b6c5-f5515237297f" containerID="e17a8b7c2f51bca1265a6eeac4702880f1a752706e7a745868ab11f1c173ebb6" exitCode=0 Jul 31 14:49:56.134258 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.134092 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md" Jul 31 14:49:56.134258 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.134126 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md" event={"ID":"cdc7b02e-48b0-4915-b6c5-f5515237297f","Type":"ContainerDied","Data":"e17a8b7c2f51bca1265a6eeac4702880f1a752706e7a745868ab11f1c173ebb6"} Jul 31 14:49:56.134258 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.134182 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md" event={"ID":"cdc7b02e-48b0-4915-b6c5-f5515237297f","Type":"ContainerDied","Data":"4eec9c24bf914d6fe2a3d01c428d2a5a0821c52e8af35e87dcef588f99870948"} Jul 31 14:49:56.134258 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.134206 2577 scope.go:117] "RemoveContainer" containerID="e17a8b7c2f51bca1265a6eeac4702880f1a752706e7a745868ab11f1c173ebb6" Jul 31 14:49:56.135578 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.135554 2577 generic.go:356] "Generic (PLEG): container finished" podID="1244ce10-52bd-4990-95cf-b99da4513a36" containerID="1f829aa64cd75ede4db1ce30bd6d7f8b96607983e2bbc25a56644ecf0ecb4b62" exitCode=0 Jul 31 14:49:56.135690 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.135604 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c" Jul 31 14:49:56.135690 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.135638 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c" event={"ID":"1244ce10-52bd-4990-95cf-b99da4513a36","Type":"ContainerDied","Data":"1f829aa64cd75ede4db1ce30bd6d7f8b96607983e2bbc25a56644ecf0ecb4b62"} Jul 31 14:49:56.135690 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.135687 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c" event={"ID":"1244ce10-52bd-4990-95cf-b99da4513a36","Type":"ContainerDied","Data":"ce089d5ef3c60d02fae82e8c5a61e08077f5b6eeb605546ce5dac3cea2aa38c4"} Jul 31 14:49:56.143623 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.143608 2577 scope.go:117] "RemoveContainer" containerID="e17a8b7c2f51bca1265a6eeac4702880f1a752706e7a745868ab11f1c173ebb6" Jul 31 14:49:56.143940 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:49:56.143915 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e17a8b7c2f51bca1265a6eeac4702880f1a752706e7a745868ab11f1c173ebb6\": container with ID starting with e17a8b7c2f51bca1265a6eeac4702880f1a752706e7a745868ab11f1c173ebb6 not found: ID does not exist" containerID="e17a8b7c2f51bca1265a6eeac4702880f1a752706e7a745868ab11f1c173ebb6" Jul 31 14:49:56.144050 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.143956 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e17a8b7c2f51bca1265a6eeac4702880f1a752706e7a745868ab11f1c173ebb6"} err="failed to get container status \"e17a8b7c2f51bca1265a6eeac4702880f1a752706e7a745868ab11f1c173ebb6\": rpc error: code = NotFound desc = could not find container \"e17a8b7c2f51bca1265a6eeac4702880f1a752706e7a745868ab11f1c173ebb6\": container with ID starting with e17a8b7c2f51bca1265a6eeac4702880f1a752706e7a745868ab11f1c173ebb6 not found: ID does not exist" Jul 31 14:49:56.144050 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.143975 2577 scope.go:117] "RemoveContainer" containerID="1f829aa64cd75ede4db1ce30bd6d7f8b96607983e2bbc25a56644ecf0ecb4b62" Jul 31 14:49:56.151582 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.151564 2577 scope.go:117] "RemoveContainer" containerID="1f829aa64cd75ede4db1ce30bd6d7f8b96607983e2bbc25a56644ecf0ecb4b62" Jul 31 14:49:56.151818 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:49:56.151798 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f829aa64cd75ede4db1ce30bd6d7f8b96607983e2bbc25a56644ecf0ecb4b62\": container with ID starting with 1f829aa64cd75ede4db1ce30bd6d7f8b96607983e2bbc25a56644ecf0ecb4b62 not found: ID does not exist" containerID="1f829aa64cd75ede4db1ce30bd6d7f8b96607983e2bbc25a56644ecf0ecb4b62" Jul 31 14:49:56.151881 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.151825 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f829aa64cd75ede4db1ce30bd6d7f8b96607983e2bbc25a56644ecf0ecb4b62"} err="failed to get container status \"1f829aa64cd75ede4db1ce30bd6d7f8b96607983e2bbc25a56644ecf0ecb4b62\": rpc error: code = NotFound desc = could not find container \"1f829aa64cd75ede4db1ce30bd6d7f8b96607983e2bbc25a56644ecf0ecb4b62\": container with ID starting with 1f829aa64cd75ede4db1ce30bd6d7f8b96607983e2bbc25a56644ecf0ecb4b62 not found: ID does not exist" Jul 31 14:49:56.156346 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.156324 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md"] Jul 31 14:49:56.160368 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.160349 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ea37d42-5465549944-jb2md"] Jul 31 14:49:56.168735 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.168713 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c"] Jul 31 14:49:56.176697 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.176676 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ea37d42-75f9464858-hkd5c"] Jul 31 14:49:56.429117 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.429043 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1244ce10-52bd-4990-95cf-b99da4513a36" path="/var/lib/kubelet/pods/1244ce10-52bd-4990-95cf-b99da4513a36/volumes" Jul 31 14:49:56.429463 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.429396 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdc7b02e-48b0-4915-b6c5-f5515237297f" path="/var/lib/kubelet/pods/cdc7b02e-48b0-4915-b6c5-f5515237297f/volumes" Jul 31 14:49:56.429682 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.429661 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4ec8184-dbfc-4635-a7e6-4769def50dda" path="/var/lib/kubelet/pods/f4ec8184-dbfc-4635-a7e6-4769def50dda/volumes" Jul 31 14:49:56.651347 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.651306 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg"] Jul 31 14:49:56.651600 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.651588 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4ec8184-dbfc-4635-a7e6-4769def50dda" containerName="revoke-keys" Jul 31 14:49:56.651697 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.651601 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4ec8184-dbfc-4635-a7e6-4769def50dda" containerName="revoke-keys" Jul 31 14:49:56.651697 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.651613 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdc7b02e-48b0-4915-b6c5-f5515237297f" containerName="payload-pre-processing" Jul 31 14:49:56.651697 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.651619 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdc7b02e-48b0-4915-b6c5-f5515237297f" containerName="payload-pre-processing" Jul 31 14:49:56.651697 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.651633 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1244ce10-52bd-4990-95cf-b99da4513a36" containerName="payload-processing" Jul 31 14:49:56.651697 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.651640 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1244ce10-52bd-4990-95cf-b99da4513a36" containerName="payload-processing" Jul 31 14:49:56.651697 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.651683 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4ec8184-dbfc-4635-a7e6-4769def50dda" containerName="revoke-keys" Jul 31 14:49:56.651697 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.651692 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdc7b02e-48b0-4915-b6c5-f5515237297f" containerName="payload-pre-processing" Jul 31 14:49:56.651987 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.651701 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1244ce10-52bd-4990-95cf-b99da4513a36" containerName="payload-processing" Jul 31 14:49:56.656002 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.655985 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.658015 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.657994 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-6abebbcb-gw-openshift-default-dockercfg-pdbxm\"" Jul 31 14:49:56.667599 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.667575 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg"] Jul 31 14:49:56.724305 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.724228 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-envoy\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.724305 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.724274 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-credential-socket\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.724305 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.724291 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-data\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.724506 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.724367 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-workload-socket\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.724506 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.724419 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-token\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.724506 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.724440 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v66xb\" (UniqueName: \"kubernetes.io/projected/e3f7f10c-48cb-43c9-9898-190ca1fc2440-kube-api-access-v66xb\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.724506 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.724493 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-workload-certs\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.724664 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.724527 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-podinfo\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.724664 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.724580 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istiod-ca-cert\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.825980 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.825943 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-workload-certs\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.825980 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.825985 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-podinfo\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.826235 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.826013 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istiod-ca-cert\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.826235 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.826045 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-envoy\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.826235 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.826075 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-credential-socket\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.826235 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.826096 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-data\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.826235 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.826124 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-workload-socket\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.826235 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.826174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-token\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.826235 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.826199 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v66xb\" (UniqueName: \"kubernetes.io/projected/e3f7f10c-48cb-43c9-9898-190ca1fc2440-kube-api-access-v66xb\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.826588 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.826417 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-workload-certs\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.826588 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.826498 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-data\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.826747 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.826724 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-workload-socket\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.826830 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.826812 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istiod-ca-cert\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.826944 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.826929 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-credential-socket\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.828307 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.828287 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-envoy\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.828526 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.828509 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-podinfo\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.833364 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.833345 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-token\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.833745 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.833722 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v66xb\" (UniqueName: \"kubernetes.io/projected/e3f7f10c-48cb-43c9-9898-190ca1fc2440-kube-api-access-v66xb\") pod \"e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:56.965597 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:56.965561 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:49:57.099205 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:57.099172 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg"] Jul 31 14:49:57.102203 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:49:57.102177 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f7f10c_48cb_43c9_9898_190ca1fc2440.slice/crio-76dd86c3f20b59d4ede67cb727db99ce4c96ce13fb537955fc7885e9463d5094 WatchSource:0}: Error finding container 76dd86c3f20b59d4ede67cb727db99ce4c96ce13fb537955fc7885e9463d5094: Status 404 returned error can't find the container with id 76dd86c3f20b59d4ede67cb727db99ce4c96ce13fb537955fc7885e9463d5094 Jul 31 14:49:57.141098 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:57.141070 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" event={"ID":"e3f7f10c-48cb-43c9-9898-190ca1fc2440","Type":"ContainerStarted","Data":"76dd86c3f20b59d4ede67cb727db99ce4c96ce13fb537955fc7885e9463d5094"} Jul 31 14:49:58.275937 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:49:58.275903 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:50:00.223144 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:00.223107 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:50:00.223410 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:00.223177 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:50:00.223410 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:00.223207 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:50:01.165280 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:01.165247 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" event={"ID":"e3f7f10c-48cb-43c9-9898-190ca1fc2440","Type":"ContainerStarted","Data":"c26e2c00054c9a2b3c1a1a3f9c92c12998ab073f617ef4b50a10094432af3966"} Jul 31 14:50:01.183024 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:01.182981 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" podStartSLOduration=2.064172268 podStartE2EDuration="5.182965958s" totalImagesPullingTime="3.11879369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:01.181126449 +0000 UTC m=+2253.217949516" watchObservedRunningTime="2026-07-31 14:50:01.182965958 +0000 UTC m=+2253.219789026" Jul 31 14:50:01.966064 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:01.966028 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:50:01.971023 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:01.971001 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:50:02.169516 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:02.169479 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:50:02.170497 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:02.170476 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:50:03.423654 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.423619 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d"] Jul 31 14:50:03.427134 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.427107 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d" Jul 31 14:50:03.428883 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.428831 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-6abebbcb\"" Jul 31 14:50:03.429009 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.428932 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-6abebbcb-dockercfg-r9kkv\"" Jul 31 14:50:03.436436 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.436411 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d"] Jul 31 14:50:03.462936 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.462905 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz"] Jul 31 14:50:03.466120 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.466103 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz" Jul 31 14:50:03.474480 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.474449 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz"] Jul 31 14:50:03.481240 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.481218 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f60ffb9-a681-4839-92f8-4c77801d8832-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d\" (UID: \"4f60ffb9-a681-4839-92f8-4c77801d8832\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d" Jul 31 14:50:03.481377 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.481323 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4dbz\" (UniqueName: \"kubernetes.io/projected/4f60ffb9-a681-4839-92f8-4c77801d8832-kube-api-access-c4dbz\") pod \"payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d\" (UID: \"4f60ffb9-a681-4839-92f8-4c77801d8832\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d" Jul 31 14:50:03.582343 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.582301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4dbz\" (UniqueName: \"kubernetes.io/projected/4f60ffb9-a681-4839-92f8-4c77801d8832-kube-api-access-c4dbz\") pod \"payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d\" (UID: \"4f60ffb9-a681-4839-92f8-4c77801d8832\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d" Jul 31 14:50:03.582520 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.582439 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f60ffb9-a681-4839-92f8-4c77801d8832-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d\" (UID: \"4f60ffb9-a681-4839-92f8-4c77801d8832\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d" Jul 31 14:50:03.582520 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.582462 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fe082c9-3679-445f-8919-818334f0dbba-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz\" (UID: \"0fe082c9-3679-445f-8919-818334f0dbba\") " pod="openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz" Jul 31 14:50:03.582520 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.582492 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm9z5\" (UniqueName: \"kubernetes.io/projected/0fe082c9-3679-445f-8919-818334f0dbba-kube-api-access-dm9z5\") pod \"payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz\" (UID: \"0fe082c9-3679-445f-8919-818334f0dbba\") " pod="openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz" Jul 31 14:50:03.583177 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.583158 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f60ffb9-a681-4839-92f8-4c77801d8832-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d\" (UID: \"4f60ffb9-a681-4839-92f8-4c77801d8832\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d" Jul 31 14:50:03.589539 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.589510 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4dbz\" (UniqueName: \"kubernetes.io/projected/4f60ffb9-a681-4839-92f8-4c77801d8832-kube-api-access-c4dbz\") pod \"payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d\" (UID: \"4f60ffb9-a681-4839-92f8-4c77801d8832\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d" Jul 31 14:50:03.683241 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.683150 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fe082c9-3679-445f-8919-818334f0dbba-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz\" (UID: \"0fe082c9-3679-445f-8919-818334f0dbba\") " pod="openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz" Jul 31 14:50:03.683241 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.683201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm9z5\" (UniqueName: \"kubernetes.io/projected/0fe082c9-3679-445f-8919-818334f0dbba-kube-api-access-dm9z5\") pod \"payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz\" (UID: \"0fe082c9-3679-445f-8919-818334f0dbba\") " pod="openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz" Jul 31 14:50:03.683991 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.683971 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fe082c9-3679-445f-8919-818334f0dbba-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz\" (UID: \"0fe082c9-3679-445f-8919-818334f0dbba\") " pod="openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz" Jul 31 14:50:03.690520 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.690488 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm9z5\" (UniqueName: \"kubernetes.io/projected/0fe082c9-3679-445f-8919-818334f0dbba-kube-api-access-dm9z5\") pod \"payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz\" (UID: \"0fe082c9-3679-445f-8919-818334f0dbba\") " pod="openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz" Jul 31 14:50:03.737807 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.737787 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d" Jul 31 14:50:03.777301 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.777050 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz" Jul 31 14:50:03.875727 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.875493 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d"] Jul 31 14:50:03.878399 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:50:03.878366 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f60ffb9_a681_4839_92f8_4c77801d8832.slice/crio-1a1a4bb34fd271bccae517d4ce01ed273a566ab29b8e00a8069ba22500bc0f92 WatchSource:0}: Error finding container 1a1a4bb34fd271bccae517d4ce01ed273a566ab29b8e00a8069ba22500bc0f92: Status 404 returned error can't find the container with id 1a1a4bb34fd271bccae517d4ce01ed273a566ab29b8e00a8069ba22500bc0f92 Jul 31 14:50:03.923313 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:03.922243 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz"] Jul 31 14:50:03.928506 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:50:03.928483 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe082c9_3679_445f_8919_818334f0dbba.slice/crio-be9063e5392cb64d645328b2eca681d7f1abd472d30b4c872ec2e60c33e942a8 WatchSource:0}: Error finding container be9063e5392cb64d645328b2eca681d7f1abd472d30b4c872ec2e60c33e942a8: Status 404 returned error can't find the container with id be9063e5392cb64d645328b2eca681d7f1abd472d30b4c872ec2e60c33e942a8 Jul 31 14:50:04.178617 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:04.178577 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d" event={"ID":"4f60ffb9-a681-4839-92f8-4c77801d8832","Type":"ContainerStarted","Data":"f5693c2e0bf9ffe03ad2a91ddef06a10cd80d4b566225772eb41c66ed08b0df4"} Jul 31 14:50:04.178774 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:04.178638 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d" event={"ID":"4f60ffb9-a681-4839-92f8-4c77801d8832","Type":"ContainerStarted","Data":"1a1a4bb34fd271bccae517d4ce01ed273a566ab29b8e00a8069ba22500bc0f92"} Jul 31 14:50:04.178774 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:04.178666 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d" Jul 31 14:50:04.180110 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:04.180083 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz" event={"ID":"0fe082c9-3679-445f-8919-818334f0dbba","Type":"ContainerStarted","Data":"cd321dcd33d170984b17df93cc8cfd4533f8cc25c438344c9021c7764bbc9702"} Jul 31 14:50:04.180220 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:04.180186 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz" Jul 31 14:50:04.180268 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:04.180226 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz" event={"ID":"0fe082c9-3679-445f-8919-818334f0dbba","Type":"ContainerStarted","Data":"be9063e5392cb64d645328b2eca681d7f1abd472d30b4c872ec2e60c33e942a8"} Jul 31 14:50:04.192434 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:04.192360 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d" podStartSLOduration=1.192345756 podStartE2EDuration="1.192345756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:04.191555307 +0000 UTC m=+2256.228378381" watchObservedRunningTime="2026-07-31 14:50:04.192345756 +0000 UTC m=+2256.229168823" Jul 31 14:50:04.206224 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:04.206184 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz" podStartSLOduration=1.206170163 podStartE2EDuration="1.206170163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:04.205194152 +0000 UTC m=+2256.242017219" watchObservedRunningTime="2026-07-31 14:50:04.206170163 +0000 UTC m=+2256.242993230" Jul 31 14:50:05.977491 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:05.977458 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:50:15.187175 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:15.187141 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz" Jul 31 14:50:15.187635 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:15.187219 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d" Jul 31 14:50:16.233590 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:16.233551 2577 generic.go:356] "Generic (PLEG): container finished" podID="e3f7f10c-48cb-43c9-9898-190ca1fc2440" containerID="c26e2c00054c9a2b3c1a1a3f9c92c12998ab073f617ef4b50a10094432af3966" exitCode=0 Jul 31 14:50:16.234049 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:16.233620 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" event={"ID":"e3f7f10c-48cb-43c9-9898-190ca1fc2440","Type":"ContainerDied","Data":"c26e2c00054c9a2b3c1a1a3f9c92c12998ab073f617ef4b50a10094432af3966"} Jul 31 14:50:16.234049 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:16.234024 2577 scope.go:117] "RemoveContainer" containerID="c26e2c00054c9a2b3c1a1a3f9c92c12998ab073f617ef4b50a10094432af3966" Jul 31 14:50:16.236005 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:16.235963 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:50:16.236122 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:16.236062 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:50:16.236122 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:16.236109 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:50:17.169630 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:17.169596 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:50:17.243918 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:17.243874 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" event={"ID":"e3f7f10c-48cb-43c9-9898-190ca1fc2440","Type":"ContainerStarted","Data":"38175a0681914d611ae61ab7b3d3fccd1ca2d98ca3e04945671fc4dd50781eb3"} Jul 31 14:50:17.966434 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:17.966395 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:50:17.967504 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:17.967481 2577 patch_prober.go:28] interesting pod/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Jul 31 14:50:17.967571 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:17.967531 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" podUID="e3f7f10c-48cb-43c9-9898-190ca1fc2440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Jul 31 14:50:18.966879 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:18.966814 2577 patch_prober.go:28] interesting pod/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Jul 31 14:50:18.967264 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:18.966898 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" podUID="e3f7f10c-48cb-43c9-9898-190ca1fc2440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Jul 31 14:50:19.966361 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:19.966320 2577 patch_prober.go:28] interesting pod/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Jul 31 14:50:19.966562 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:19.966384 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" podUID="e3f7f10c-48cb-43c9-9898-190ca1fc2440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Jul 31 14:50:20.966372 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:20.966331 2577 patch_prober.go:28] interesting pod/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Jul 31 14:50:20.966754 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:20.966387 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" podUID="e3f7f10c-48cb-43c9-9898-190ca1fc2440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Jul 31 14:50:21.966864 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:21.966804 2577 patch_prober.go:28] interesting pod/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Jul 31 14:50:21.967232 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:21.966885 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" podUID="e3f7f10c-48cb-43c9-9898-190ca1fc2440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Jul 31 14:50:22.970583 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:22.970546 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:50:22.971066 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:22.970921 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:50:22.971505 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:22.971485 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:50:34.176952 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.176920 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz"] Jul 31 14:50:34.177343 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.177237 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz" podUID="0fe082c9-3679-445f-8919-818334f0dbba" containerName="payload-processing" containerID="cri-o://cd321dcd33d170984b17df93cc8cfd4533f8cc25c438344c9021c7764bbc9702" gracePeriod=30 Jul 31 14:50:34.181386 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.181345 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d"] Jul 31 14:50:34.181572 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.181546 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d" podUID="4f60ffb9-a681-4839-92f8-4c77801d8832" containerName="payload-pre-processing" containerID="cri-o://f5693c2e0bf9ffe03ad2a91ddef06a10cd80d4b566225772eb41c66ed08b0df4" gracePeriod=30 Jul 31 14:50:34.325325 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.325285 2577 generic.go:356] "Generic (PLEG): container finished" podID="4f60ffb9-a681-4839-92f8-4c77801d8832" containerID="f5693c2e0bf9ffe03ad2a91ddef06a10cd80d4b566225772eb41c66ed08b0df4" exitCode=0 Jul 31 14:50:34.325505 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.325391 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d" event={"ID":"4f60ffb9-a681-4839-92f8-4c77801d8832","Type":"ContainerDied","Data":"f5693c2e0bf9ffe03ad2a91ddef06a10cd80d4b566225772eb41c66ed08b0df4"} Jul 31 14:50:34.327211 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.327182 2577 generic.go:356] "Generic (PLEG): container finished" podID="0fe082c9-3679-445f-8919-818334f0dbba" containerID="cd321dcd33d170984b17df93cc8cfd4533f8cc25c438344c9021c7764bbc9702" exitCode=0 Jul 31 14:50:34.327412 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.327246 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz" event={"ID":"0fe082c9-3679-445f-8919-818334f0dbba","Type":"ContainerDied","Data":"cd321dcd33d170984b17df93cc8cfd4533f8cc25c438344c9021c7764bbc9702"} Jul 31 14:50:34.453221 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.453124 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg"] Jul 31 14:50:34.453644 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.453612 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" podUID="e3f7f10c-48cb-43c9-9898-190ca1fc2440" containerName="istio-proxy" containerID="cri-o://38175a0681914d611ae61ab7b3d3fccd1ca2d98ca3e04945671fc4dd50781eb3" gracePeriod=30 Jul 31 14:50:34.517396 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.517362 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d" Jul 31 14:50:34.520449 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.520423 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz" Jul 31 14:50:34.565897 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.565871 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fe082c9-3679-445f-8919-818334f0dbba-plugins-config-volume\") pod \"0fe082c9-3679-445f-8919-818334f0dbba\" (UID: \"0fe082c9-3679-445f-8919-818334f0dbba\") " Jul 31 14:50:34.566093 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.565925 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4dbz\" (UniqueName: \"kubernetes.io/projected/4f60ffb9-a681-4839-92f8-4c77801d8832-kube-api-access-c4dbz\") pod \"4f60ffb9-a681-4839-92f8-4c77801d8832\" (UID: \"4f60ffb9-a681-4839-92f8-4c77801d8832\") " Jul 31 14:50:34.566093 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.565963 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm9z5\" (UniqueName: \"kubernetes.io/projected/0fe082c9-3679-445f-8919-818334f0dbba-kube-api-access-dm9z5\") pod \"0fe082c9-3679-445f-8919-818334f0dbba\" (UID: \"0fe082c9-3679-445f-8919-818334f0dbba\") " Jul 31 14:50:34.566093 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.566040 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f60ffb9-a681-4839-92f8-4c77801d8832-plugins-config-volume\") pod \"4f60ffb9-a681-4839-92f8-4c77801d8832\" (UID: \"4f60ffb9-a681-4839-92f8-4c77801d8832\") " Jul 31 14:50:34.566276 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.566249 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe082c9-3679-445f-8919-818334f0dbba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0fe082c9-3679-445f-8919-818334f0dbba" (UID: "0fe082c9-3679-445f-8919-818334f0dbba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:50:34.566501 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.566474 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f60ffb9-a681-4839-92f8-4c77801d8832-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f60ffb9-a681-4839-92f8-4c77801d8832" (UID: "4f60ffb9-a681-4839-92f8-4c77801d8832"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:50:34.568070 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.568038 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f60ffb9-a681-4839-92f8-4c77801d8832-kube-api-access-c4dbz" (OuterVolumeSpecName: "kube-api-access-c4dbz") pod "4f60ffb9-a681-4839-92f8-4c77801d8832" (UID: "4f60ffb9-a681-4839-92f8-4c77801d8832"). InnerVolumeSpecName "kube-api-access-c4dbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:50:34.568204 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.568187 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe082c9-3679-445f-8919-818334f0dbba-kube-api-access-dm9z5" (OuterVolumeSpecName: "kube-api-access-dm9z5") pod "0fe082c9-3679-445f-8919-818334f0dbba" (UID: "0fe082c9-3679-445f-8919-818334f0dbba"). InnerVolumeSpecName "kube-api-access-dm9z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:50:34.666798 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.666766 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f60ffb9-a681-4839-92f8-4c77801d8832-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:50:34.666798 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.666797 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fe082c9-3679-445f-8919-818334f0dbba-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:50:34.667019 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.666809 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4dbz\" (UniqueName: \"kubernetes.io/projected/4f60ffb9-a681-4839-92f8-4c77801d8832-kube-api-access-c4dbz\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:50:34.667019 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:34.666823 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm9z5\" (UniqueName: \"kubernetes.io/projected/0fe082c9-3679-445f-8919-818334f0dbba-kube-api-access-dm9z5\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:50:35.244405 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.244375 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25"] Jul 31 14:50:35.244758 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.244695 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f60ffb9-a681-4839-92f8-4c77801d8832" containerName="payload-pre-processing" Jul 31 14:50:35.244758 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.244707 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f60ffb9-a681-4839-92f8-4c77801d8832" containerName="payload-pre-processing" Jul 31 14:50:35.244758 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.244717 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fe082c9-3679-445f-8919-818334f0dbba" containerName="payload-processing" Jul 31 14:50:35.244758 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.244723 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fe082c9-3679-445f-8919-818334f0dbba" containerName="payload-processing" Jul 31 14:50:35.244893 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.244796 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fe082c9-3679-445f-8919-818334f0dbba" containerName="payload-processing" Jul 31 14:50:35.244893 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.244810 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f60ffb9-a681-4839-92f8-4c77801d8832" containerName="payload-pre-processing" Jul 31 14:50:35.247901 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.247885 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.249710 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.249692 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6c458921-openshift-default-dockercfg-lqr9q\"" Jul 31 14:50:35.259658 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.259636 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25"] Jul 31 14:50:35.272969 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.272944 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-credential-socket\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.273080 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.272975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-data\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.273080 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.273000 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-podinfo\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.273080 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.273021 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbjlx\" (UniqueName: \"kubernetes.io/projected/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-kube-api-access-lbjlx\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.273213 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.273081 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-token\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.273213 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.273102 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istiod-ca-cert\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.273213 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.273171 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-workload-certs\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.273328 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.273237 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-envoy\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.273328 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.273269 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-workload-socket\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.334984 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.334952 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz" Jul 31 14:50:35.335144 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.334981 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz" event={"ID":"0fe082c9-3679-445f-8919-818334f0dbba","Type":"ContainerDied","Data":"be9063e5392cb64d645328b2eca681d7f1abd472d30b4c872ec2e60c33e942a8"} Jul 31 14:50:35.335144 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.335040 2577 scope.go:117] "RemoveContainer" containerID="cd321dcd33d170984b17df93cc8cfd4533f8cc25c438344c9021c7764bbc9702" Jul 31 14:50:35.336491 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.336464 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d" Jul 31 14:50:35.336583 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.336464 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d" event={"ID":"4f60ffb9-a681-4839-92f8-4c77801d8832","Type":"ContainerDied","Data":"1a1a4bb34fd271bccae517d4ce01ed273a566ab29b8e00a8069ba22500bc0f92"} Jul 31 14:50:35.344967 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.344669 2577 scope.go:117] "RemoveContainer" containerID="f5693c2e0bf9ffe03ad2a91ddef06a10cd80d4b566225772eb41c66ed08b0df4" Jul 31 14:50:35.362011 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.361983 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz"] Jul 31 14:50:35.365923 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.365898 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6abebbcb-df69d558f-trnnz"] Jul 31 14:50:35.374097 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.374076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-envoy\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.374211 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.374107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-workload-socket\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.374211 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.374135 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-credential-socket\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.374211 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.374154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-data\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.374211 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.374178 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-podinfo\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.374211 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.374201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbjlx\" (UniqueName: \"kubernetes.io/projected/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-kube-api-access-lbjlx\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.374462 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.374239 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-token\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.374462 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.374354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istiod-ca-cert\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.374462 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.374387 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-workload-certs\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.374614 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.374599 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-data\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.374663 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.374636 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-credential-socket\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.374799 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.374768 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-workload-certs\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.374799 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.374784 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-workload-socket\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.375072 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.375046 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istiod-ca-cert\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.377104 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.377076 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-envoy\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.377438 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.377421 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-podinfo\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.377712 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.377695 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d"] Jul 31 14:50:35.386439 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.386420 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6abebbcb-56db7cb688-v2h4d"] Jul 31 14:50:35.388322 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.388294 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbjlx\" (UniqueName: \"kubernetes.io/projected/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-kube-api-access-lbjlx\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.388539 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.388521 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-token\") pod \"e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.557808 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.557728 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:35.684891 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.684866 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25"] Jul 31 14:50:35.687102 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:50:35.687075 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec56faf_cd5c_4cc3_b02c_b09fadae0e9c.slice/crio-9638bb5dab701d006f93bf101e90f85dc6fe2d825d07675dc412ab32358ef77c WatchSource:0}: Error finding container 9638bb5dab701d006f93bf101e90f85dc6fe2d825d07675dc412ab32358ef77c: Status 404 returned error can't find the container with id 9638bb5dab701d006f93bf101e90f85dc6fe2d825d07675dc412ab32358ef77c Jul 31 14:50:35.689244 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.689214 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:50:35.689330 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.689284 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:50:35.689389 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:35.689331 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:50:36.348315 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:36.348274 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" event={"ID":"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c","Type":"ContainerStarted","Data":"6c4bb8f3589c13e076a0ed53f96e6009c443110942c4e5d625a2dc91d0ba814a"} Jul 31 14:50:36.348759 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:36.348368 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" event={"ID":"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c","Type":"ContainerStarted","Data":"9638bb5dab701d006f93bf101e90f85dc6fe2d825d07675dc412ab32358ef77c"} Jul 31 14:50:36.365361 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:36.365197 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" podStartSLOduration=1.36518117 podStartE2EDuration="1.36518117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:36.364938539 +0000 UTC m=+2288.401761606" watchObservedRunningTime="2026-07-31 14:50:36.36518117 +0000 UTC m=+2288.402004237" Jul 31 14:50:36.429306 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:36.429279 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fe082c9-3679-445f-8919-818334f0dbba" path="/var/lib/kubelet/pods/0fe082c9-3679-445f-8919-818334f0dbba/volumes" Jul 31 14:50:36.429647 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:36.429634 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f60ffb9-a681-4839-92f8-4c77801d8832" path="/var/lib/kubelet/pods/4f60ffb9-a681-4839-92f8-4c77801d8832/volumes" Jul 31 14:50:36.558254 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:36.558223 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:36.563042 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:36.563018 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:37.076489 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:37.076458 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:50:37.354942 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:37.354835 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:37.355763 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:37.355745 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:37.972757 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:37.972721 2577 patch_prober.go:28] interesting pod/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 31 14:50:37.972940 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:37.972780 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" podUID="e3f7f10c-48cb-43c9-9898-190ca1fc2440" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 14:50:39.420505 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.420477 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd"] Jul 31 14:50:39.425280 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.425255 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd" Jul 31 14:50:39.427161 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.427132 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6c458921\"" Jul 31 14:50:39.427271 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.427182 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6c458921-dockercfg-xjxcb\"" Jul 31 14:50:39.434330 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.434295 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd"] Jul 31 14:50:39.511072 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.511040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62-plugins-config-volume\") pod \"payload-processing-e2e-ait-6c458921-b6956466d-7tfgd\" (UID: \"bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62\") " pod="openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd" Jul 31 14:50:39.511244 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.511196 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm2xp\" (UniqueName: \"kubernetes.io/projected/bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62-kube-api-access-jm2xp\") pod \"payload-processing-e2e-ait-6c458921-b6956466d-7tfgd\" (UID: \"bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62\") " pod="openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd" Jul 31 14:50:39.612737 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.612706 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62-plugins-config-volume\") pod \"payload-processing-e2e-ait-6c458921-b6956466d-7tfgd\" (UID: \"bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62\") " pod="openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd" Jul 31 14:50:39.612909 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.612865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2xp\" (UniqueName: \"kubernetes.io/projected/bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62-kube-api-access-jm2xp\") pod \"payload-processing-e2e-ait-6c458921-b6956466d-7tfgd\" (UID: \"bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62\") " pod="openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd" Jul 31 14:50:39.613500 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.613475 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62-plugins-config-volume\") pod \"payload-processing-e2e-ait-6c458921-b6956466d-7tfgd\" (UID: \"bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62\") " pod="openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd" Jul 31 14:50:39.620104 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.620081 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2xp\" (UniqueName: \"kubernetes.io/projected/bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62-kube-api-access-jm2xp\") pod \"payload-processing-e2e-ait-6c458921-b6956466d-7tfgd\" (UID: \"bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62\") " pod="openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd" Jul 31 14:50:39.713208 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.713184 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:50:39.736264 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.736240 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd" Jul 31 14:50:39.814068 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.814035 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-data\") pod \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " Jul 31 14:50:39.814199 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.814131 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-token\") pod \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " Jul 31 14:50:39.814344 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.814324 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3f7f10c-48cb-43c9-9898-190ca1fc2440" (UID: "e3f7f10c-48cb-43c9-9898-190ca1fc2440"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:50:39.814550 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.814533 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v66xb\" (UniqueName: \"kubernetes.io/projected/e3f7f10c-48cb-43c9-9898-190ca1fc2440-kube-api-access-v66xb\") pod \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " Jul 31 14:50:39.814597 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.814583 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-envoy\") pod \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " Jul 31 14:50:39.814670 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.814657 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-workload-certs\") pod \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " Jul 31 14:50:39.814734 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.814721 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-credential-socket\") pod \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " Jul 31 14:50:39.814766 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.814752 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istiod-ca-cert\") pod \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " Jul 31 14:50:39.814864 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.814791 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-workload-socket\") pod \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " Jul 31 14:50:39.814864 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.814827 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-podinfo\") pod \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\" (UID: \"e3f7f10c-48cb-43c9-9898-190ca1fc2440\") " Jul 31 14:50:39.815252 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.815059 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3f7f10c-48cb-43c9-9898-190ca1fc2440" (UID: "e3f7f10c-48cb-43c9-9898-190ca1fc2440"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:50:39.815252 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.815229 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-credential-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:50:39.815252 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.815248 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-data\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:50:39.815462 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.815214 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3f7f10c-48cb-43c9-9898-190ca1fc2440" (UID: "e3f7f10c-48cb-43c9-9898-190ca1fc2440"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:50:39.815462 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.815277 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3f7f10c-48cb-43c9-9898-190ca1fc2440" (UID: "e3f7f10c-48cb-43c9-9898-190ca1fc2440"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:50:39.815462 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.815351 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3f7f10c-48cb-43c9-9898-190ca1fc2440" (UID: "e3f7f10c-48cb-43c9-9898-190ca1fc2440"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:50:39.816527 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.816487 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f7f10c-48cb-43c9-9898-190ca1fc2440-kube-api-access-v66xb" (OuterVolumeSpecName: "kube-api-access-v66xb") pod "e3f7f10c-48cb-43c9-9898-190ca1fc2440" (UID: "e3f7f10c-48cb-43c9-9898-190ca1fc2440"). InnerVolumeSpecName "kube-api-access-v66xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:50:39.816886 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.816863 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3f7f10c-48cb-43c9-9898-190ca1fc2440" (UID: "e3f7f10c-48cb-43c9-9898-190ca1fc2440"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:50:39.817067 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.817049 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3f7f10c-48cb-43c9-9898-190ca1fc2440" (UID: "e3f7f10c-48cb-43c9-9898-190ca1fc2440"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:50:39.817155 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.817140 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3f7f10c-48cb-43c9-9898-190ca1fc2440" (UID: "e3f7f10c-48cb-43c9-9898-190ca1fc2440"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 14:50:39.870980 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.870951 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd"] Jul 31 14:50:39.874288 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:50:39.874264 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2aade9_a9e1_4ee9_a7c7_aec8f7d80e62.slice/crio-093a4db766710db73719542b11223eda11720e234250f4b9da92e3e51475c074 WatchSource:0}: Error finding container 093a4db766710db73719542b11223eda11720e234250f4b9da92e3e51475c074: Status 404 returned error can't find the container with id 093a4db766710db73719542b11223eda11720e234250f4b9da92e3e51475c074 Jul 31 14:50:39.915609 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.915588 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-workload-certs\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:50:39.915697 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.915611 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istiod-ca-cert\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:50:39.915697 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.915620 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-workload-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:50:39.915697 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.915629 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-podinfo\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:50:39.915697 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.915638 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-token\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:50:39.915697 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.915646 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v66xb\" (UniqueName: \"kubernetes.io/projected/e3f7f10c-48cb-43c9-9898-190ca1fc2440-kube-api-access-v66xb\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:50:39.915697 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:39.915654 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f7f10c-48cb-43c9-9898-190ca1fc2440-istio-envoy\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:50:40.367730 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:40.367702 2577 generic.go:356] "Generic (PLEG): container finished" podID="e3f7f10c-48cb-43c9-9898-190ca1fc2440" containerID="38175a0681914d611ae61ab7b3d3fccd1ca2d98ca3e04945671fc4dd50781eb3" exitCode=0 Jul 31 14:50:40.367918 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:40.367754 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" Jul 31 14:50:40.367918 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:40.367790 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" event={"ID":"e3f7f10c-48cb-43c9-9898-190ca1fc2440","Type":"ContainerDied","Data":"38175a0681914d611ae61ab7b3d3fccd1ca2d98ca3e04945671fc4dd50781eb3"} Jul 31 14:50:40.367918 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:40.367869 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg" event={"ID":"e3f7f10c-48cb-43c9-9898-190ca1fc2440","Type":"ContainerDied","Data":"76dd86c3f20b59d4ede67cb727db99ce4c96ce13fb537955fc7885e9463d5094"} Jul 31 14:50:40.367918 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:40.367899 2577 scope.go:117] "RemoveContainer" containerID="38175a0681914d611ae61ab7b3d3fccd1ca2d98ca3e04945671fc4dd50781eb3" Jul 31 14:50:40.369279 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:40.369255 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd" event={"ID":"bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62","Type":"ContainerStarted","Data":"11b16ce791d44ae4a70a406156698de3b464793109c0972b4f96cc0a3ffe7678"} Jul 31 14:50:40.369386 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:40.369303 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd" event={"ID":"bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62","Type":"ContainerStarted","Data":"093a4db766710db73719542b11223eda11720e234250f4b9da92e3e51475c074"} Jul 31 14:50:40.369386 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:40.369344 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd" Jul 31 14:50:40.376708 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:40.376687 2577 scope.go:117] "RemoveContainer" containerID="c26e2c00054c9a2b3c1a1a3f9c92c12998ab073f617ef4b50a10094432af3966" Jul 31 14:50:40.384253 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:40.384238 2577 scope.go:117] "RemoveContainer" containerID="38175a0681914d611ae61ab7b3d3fccd1ca2d98ca3e04945671fc4dd50781eb3" Jul 31 14:50:40.384484 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:50:40.384460 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38175a0681914d611ae61ab7b3d3fccd1ca2d98ca3e04945671fc4dd50781eb3\": container with ID starting with 38175a0681914d611ae61ab7b3d3fccd1ca2d98ca3e04945671fc4dd50781eb3 not found: ID does not exist" containerID="38175a0681914d611ae61ab7b3d3fccd1ca2d98ca3e04945671fc4dd50781eb3" Jul 31 14:50:40.384542 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:40.384506 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38175a0681914d611ae61ab7b3d3fccd1ca2d98ca3e04945671fc4dd50781eb3"} err="failed to get container status \"38175a0681914d611ae61ab7b3d3fccd1ca2d98ca3e04945671fc4dd50781eb3\": rpc error: code = NotFound desc = could not find container \"38175a0681914d611ae61ab7b3d3fccd1ca2d98ca3e04945671fc4dd50781eb3\": container with ID starting with 38175a0681914d611ae61ab7b3d3fccd1ca2d98ca3e04945671fc4dd50781eb3 not found: ID does not exist" Jul 31 14:50:40.384542 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:40.384529 2577 scope.go:117] "RemoveContainer" containerID="c26e2c00054c9a2b3c1a1a3f9c92c12998ab073f617ef4b50a10094432af3966" Jul 31 14:50:40.384809 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:50:40.384781 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c26e2c00054c9a2b3c1a1a3f9c92c12998ab073f617ef4b50a10094432af3966\": container with ID starting with c26e2c00054c9a2b3c1a1a3f9c92c12998ab073f617ef4b50a10094432af3966 not found: ID does not exist" containerID="c26e2c00054c9a2b3c1a1a3f9c92c12998ab073f617ef4b50a10094432af3966" Jul 31 14:50:40.385076 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:40.384816 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c26e2c00054c9a2b3c1a1a3f9c92c12998ab073f617ef4b50a10094432af3966"} err="failed to get container status \"c26e2c00054c9a2b3c1a1a3f9c92c12998ab073f617ef4b50a10094432af3966\": rpc error: code = NotFound desc = could not find container \"c26e2c00054c9a2b3c1a1a3f9c92c12998ab073f617ef4b50a10094432af3966\": container with ID starting with c26e2c00054c9a2b3c1a1a3f9c92c12998ab073f617ef4b50a10094432af3966 not found: ID does not exist" Jul 31 14:50:40.386630 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:40.386595 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd" podStartSLOduration=1.3865826509999999 podStartE2EDuration="1.386582651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:40.385462731 +0000 UTC m=+2292.422285802" watchObservedRunningTime="2026-07-31 14:50:40.386582651 +0000 UTC m=+2292.423405717" Jul 31 14:50:40.398319 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:40.398293 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg"] Jul 31 14:50:40.402037 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:40.402013 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-6abebbcb-gw-openshift-default-75869dcd5b-k2dcg"] Jul 31 14:50:40.434713 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:40.434680 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3f7f10c-48cb-43c9-9898-190ca1fc2440" path="/var/lib/kubelet/pods/e3f7f10c-48cb-43c9-9898-190ca1fc2440/volumes" Jul 31 14:50:45.475107 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:45.475077 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:50:51.378018 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:51.377988 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd" Jul 31 14:50:51.425690 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:51.425621 2577 generic.go:356] "Generic (PLEG): container finished" podID="3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" containerID="6c4bb8f3589c13e076a0ed53f96e6009c443110942c4e5d625a2dc91d0ba814a" exitCode=0 Jul 31 14:50:51.425807 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:51.425685 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" event={"ID":"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c","Type":"ContainerDied","Data":"6c4bb8f3589c13e076a0ed53f96e6009c443110942c4e5d625a2dc91d0ba814a"} Jul 31 14:50:51.426068 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:51.426055 2577 scope.go:117] "RemoveContainer" containerID="6c4bb8f3589c13e076a0ed53f96e6009c443110942c4e5d625a2dc91d0ba814a" Jul 31 14:50:51.427121 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:51.427080 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:50:51.427199 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:51.427176 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:50:51.427242 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:51.427223 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:50:52.355525 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:52.355483 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:52.435400 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:52.435351 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" event={"ID":"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c","Type":"ContainerStarted","Data":"9c8cafd3b39a82414cd657cbd318a58682fe6dd0bb280df44cc62431024e4e09"} Jul 31 14:50:52.558810 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:52.558777 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:52.559984 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:52.559957 2577 patch_prober.go:28] interesting pod/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 31 14:50:52.560130 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:52.560032 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" podUID="3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 31 14:50:53.558328 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:53.558295 2577 patch_prober.go:28] interesting pod/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 31 14:50:53.558700 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:53.558350 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" podUID="3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 31 14:50:54.558529 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:54.558492 2577 patch_prober.go:28] interesting pod/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 31 14:50:54.558919 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:54.558557 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" podUID="3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 31 14:50:55.558124 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:55.558088 2577 patch_prober.go:28] interesting pod/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 31 14:50:55.558292 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:55.558162 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" podUID="3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 31 14:50:56.558328 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:56.558289 2577 patch_prober.go:28] interesting pod/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 31 14:50:56.558731 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:56.558346 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" podUID="3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 31 14:50:57.562590 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:57.562558 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:57.562976 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:57.562803 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:50:57.563552 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:50:57.563537 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:51:03.568119 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:03.568086 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c458921-pbnk7"] Jul 31 14:51:03.568513 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:03.568370 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f7f10c-48cb-43c9-9898-190ca1fc2440" containerName="istio-proxy" Jul 31 14:51:03.568513 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:03.568380 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f7f10c-48cb-43c9-9898-190ca1fc2440" containerName="istio-proxy" Jul 31 14:51:03.568513 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:03.568449 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f7f10c-48cb-43c9-9898-190ca1fc2440" containerName="istio-proxy" Jul 31 14:51:03.571536 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:03.571520 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c458921-pbnk7" Jul 31 14:51:03.573281 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:03.573259 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 14:51:03.573388 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:03.573259 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\"" Jul 31 14:51:03.577375 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:03.577352 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c458921-pbnk7"] Jul 31 14:51:03.733688 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:03.733637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94026054-4342-4947-add3-1a57bad78e42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6c458921-pbnk7\" (UID: \"94026054-4342-4947-add3-1a57bad78e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c458921-pbnk7" Jul 31 14:51:03.835083 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:03.835001 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94026054-4342-4947-add3-1a57bad78e42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6c458921-pbnk7\" (UID: \"94026054-4342-4947-add3-1a57bad78e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c458921-pbnk7" Jul 31 14:51:03.835637 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:03.835610 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94026054-4342-4947-add3-1a57bad78e42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6c458921-pbnk7\" (UID: \"94026054-4342-4947-add3-1a57bad78e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c458921-pbnk7" Jul 31 14:51:03.882464 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:03.882439 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c458921-pbnk7" Jul 31 14:51:04.007910 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:04.007876 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c458921-pbnk7"] Jul 31 14:51:04.011134 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:51:04.011108 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94026054_4342_4947_add3_1a57bad78e42.slice/crio-b62d88755dcae7b4295db2e6fb1aef1d2d3eb9d1fbfd87da9e0c903cc5f9505d WatchSource:0}: Error finding container b62d88755dcae7b4295db2e6fb1aef1d2d3eb9d1fbfd87da9e0c903cc5f9505d: Status 404 returned error can't find the container with id b62d88755dcae7b4295db2e6fb1aef1d2d3eb9d1fbfd87da9e0c903cc5f9505d Jul 31 14:51:04.496712 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:04.496618 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c458921-pbnk7" event={"ID":"94026054-4342-4947-add3-1a57bad78e42","Type":"ContainerStarted","Data":"8444e86bf4e26713f2202444af8132293712a92975125c7bf439ab0368fbffde"} Jul 31 14:51:04.496712 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:04.496674 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c458921-pbnk7" event={"ID":"94026054-4342-4947-add3-1a57bad78e42","Type":"ContainerStarted","Data":"b62d88755dcae7b4295db2e6fb1aef1d2d3eb9d1fbfd87da9e0c903cc5f9505d"} Jul 31 14:51:04.510704 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:04.510661 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c458921-pbnk7" podStartSLOduration=1.5106471099999998 podStartE2EDuration="1.51064711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:51:04.508942613 +0000 UTC m=+2316.545765681" watchObservedRunningTime="2026-07-31 14:51:04.51064711 +0000 UTC m=+2316.547470202" Jul 31 14:51:05.502600 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:05.502568 2577 generic.go:356] "Generic (PLEG): container finished" podID="94026054-4342-4947-add3-1a57bad78e42" containerID="8444e86bf4e26713f2202444af8132293712a92975125c7bf439ab0368fbffde" exitCode=0 Jul 31 14:51:05.502984 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:05.502662 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c458921-pbnk7" event={"ID":"94026054-4342-4947-add3-1a57bad78e42","Type":"ContainerDied","Data":"8444e86bf4e26713f2202444af8132293712a92975125c7bf439ab0368fbffde"} Jul 31 14:51:06.621386 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:06.621361 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c458921-pbnk7" Jul 31 14:51:06.760654 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:06.760567 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94026054-4342-4947-add3-1a57bad78e42-maas-api-service-ca\") pod \"94026054-4342-4947-add3-1a57bad78e42\" (UID: \"94026054-4342-4947-add3-1a57bad78e42\") " Jul 31 14:51:06.760899 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:06.760876 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94026054-4342-4947-add3-1a57bad78e42-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94026054-4342-4947-add3-1a57bad78e42" (UID: "94026054-4342-4947-add3-1a57bad78e42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:51:06.861129 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:06.861100 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94026054-4342-4947-add3-1a57bad78e42-maas-api-service-ca\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:07.513102 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:07.513071 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c458921-pbnk7" Jul 31 14:51:07.513281 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:07.513106 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c458921-pbnk7" event={"ID":"94026054-4342-4947-add3-1a57bad78e42","Type":"ContainerDied","Data":"b62d88755dcae7b4295db2e6fb1aef1d2d3eb9d1fbfd87da9e0c903cc5f9505d"} Jul 31 14:51:07.513281 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:07.513150 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b62d88755dcae7b4295db2e6fb1aef1d2d3eb9d1fbfd87da9e0c903cc5f9505d" Jul 31 14:51:09.309204 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:09.309168 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd"] Jul 31 14:51:09.309602 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:09.309395 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd" podUID="bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62" containerName="payload-processing" containerID="cri-o://11b16ce791d44ae4a70a406156698de3b464793109c0972b4f96cc0a3ffe7678" gracePeriod=30 Jul 31 14:51:09.529294 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:09.529259 2577 generic.go:356] "Generic (PLEG): container finished" podID="bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62" containerID="11b16ce791d44ae4a70a406156698de3b464793109c0972b4f96cc0a3ffe7678" exitCode=0 Jul 31 14:51:09.529455 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:09.529311 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd" event={"ID":"bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62","Type":"ContainerDied","Data":"11b16ce791d44ae4a70a406156698de3b464793109c0972b4f96cc0a3ffe7678"} Jul 31 14:51:09.548677 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:09.548653 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd" Jul 31 14:51:09.552455 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:09.552432 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c458921-pbnk7"] Jul 31 14:51:09.556425 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:09.556398 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c458921-pbnk7"] Jul 31 14:51:09.684854 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:09.684800 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62-plugins-config-volume\") pod \"bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62\" (UID: \"bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62\") " Jul 31 14:51:09.684854 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:09.684836 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jm2xp\" (UniqueName: \"kubernetes.io/projected/bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62-kube-api-access-jm2xp\") pod \"bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62\" (UID: \"bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62\") " Jul 31 14:51:09.685218 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:09.685191 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62" (UID: "bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:51:09.687055 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:09.687029 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62-kube-api-access-jm2xp" (OuterVolumeSpecName: "kube-api-access-jm2xp") pod "bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62" (UID: "bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62"). InnerVolumeSpecName "kube-api-access-jm2xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:51:09.785879 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:09.785835 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:09.785879 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:09.785874 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm2xp\" (UniqueName: \"kubernetes.io/projected/bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62-kube-api-access-jm2xp\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:10.429536 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:10.429503 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94026054-4342-4947-add3-1a57bad78e42" path="/var/lib/kubelet/pods/94026054-4342-4947-add3-1a57bad78e42/volumes" Jul 31 14:51:10.535857 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:10.535813 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd" Jul 31 14:51:10.536036 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:10.535830 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd" event={"ID":"bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62","Type":"ContainerDied","Data":"093a4db766710db73719542b11223eda11720e234250f4b9da92e3e51475c074"} Jul 31 14:51:10.536036 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:10.535912 2577 scope.go:117] "RemoveContainer" containerID="11b16ce791d44ae4a70a406156698de3b464793109c0972b4f96cc0a3ffe7678" Jul 31 14:51:10.550507 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:10.550487 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd"] Jul 31 14:51:10.557652 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:10.557634 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6c458921-b6956466d-7tfgd"] Jul 31 14:51:10.785659 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:10.785584 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25"] Jul 31 14:51:10.785827 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:10.785801 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" podUID="3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" containerName="istio-proxy" containerID="cri-o://9c8cafd3b39a82414cd657cbd318a58682fe6dd0bb280df44cc62431024e4e09" gracePeriod=30 Jul 31 14:51:11.590880 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.590833 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl"] Jul 31 14:51:11.591351 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.591294 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f7f10c-48cb-43c9-9898-190ca1fc2440" containerName="istio-proxy" Jul 31 14:51:11.591351 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.591311 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f7f10c-48cb-43c9-9898-190ca1fc2440" containerName="istio-proxy" Jul 31 14:51:11.591351 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.591334 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94026054-4342-4947-add3-1a57bad78e42" containerName="revoke-keys" Jul 31 14:51:11.591351 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.591345 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="94026054-4342-4947-add3-1a57bad78e42" containerName="revoke-keys" Jul 31 14:51:11.591561 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.591356 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62" containerName="payload-processing" Jul 31 14:51:11.591561 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.591365 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62" containerName="payload-processing" Jul 31 14:51:11.591561 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.591448 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="94026054-4342-4947-add3-1a57bad78e42" containerName="revoke-keys" Jul 31 14:51:11.591561 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.591463 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f7f10c-48cb-43c9-9898-190ca1fc2440" containerName="istio-proxy" Jul 31 14:51:11.591561 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.591474 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62" containerName="payload-processing" Jul 31 14:51:11.596213 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.596196 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.598206 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.598187 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3187acc6-openshift-default-dockercfg-76kbv\"" Jul 31 14:51:11.604670 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.604648 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl"] Jul 31 14:51:11.701778 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.701729 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-data\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.701977 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.701855 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-credential-socket\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.701977 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.701927 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-workload-certs\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.702109 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.702027 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-podinfo\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.702109 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.702067 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istiod-ca-cert\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.702204 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.702131 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffk7t\" (UniqueName: \"kubernetes.io/projected/5f7cb9b3-6905-485d-b735-1d6d5a83d234-kube-api-access-ffk7t\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.702204 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.702159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-envoy\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.702278 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.702238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-workload-socket\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.702322 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.702276 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-token\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.802701 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.802664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-workload-certs\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.802894 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.802712 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-podinfo\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.802894 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.802730 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istiod-ca-cert\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.802894 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.802767 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffk7t\" (UniqueName: \"kubernetes.io/projected/5f7cb9b3-6905-485d-b735-1d6d5a83d234-kube-api-access-ffk7t\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.802894 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.802791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-envoy\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.802894 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.802821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-workload-socket\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.802894 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.802870 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-token\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.803209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.802927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-data\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.803209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.802967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-credential-socket\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.803209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.803126 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-workload-certs\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.803399 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.803368 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-credential-socket\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.803525 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.803404 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-workload-socket\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.803525 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.803485 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-data\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.803636 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.803573 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istiod-ca-cert\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.805055 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.805029 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-envoy\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.805341 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.805325 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-podinfo\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.813957 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.813938 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-token\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.814244 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.814227 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffk7t\" (UniqueName: \"kubernetes.io/projected/5f7cb9b3-6905-485d-b735-1d6d5a83d234-kube-api-access-ffk7t\") pod \"e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:11.907057 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:11.907028 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:12.041603 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:12.041559 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl"] Jul 31 14:51:12.044656 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:51:12.044623 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7cb9b3_6905_485d_b735_1d6d5a83d234.slice/crio-ff8542c110f73da9454fe1a536896bc834ad2aa944824f7e3473b162d74faf3f WatchSource:0}: Error finding container ff8542c110f73da9454fe1a536896bc834ad2aa944824f7e3473b162d74faf3f: Status 404 returned error can't find the container with id ff8542c110f73da9454fe1a536896bc834ad2aa944824f7e3473b162d74faf3f Jul 31 14:51:12.047182 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:12.047150 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:51:12.047283 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:12.047230 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:51:12.047344 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:12.047299 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:51:12.290696 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:12.290623 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:51:12.430775 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:12.430737 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62" path="/var/lib/kubelet/pods/bf2aade9-a9e1-4ee9-a7c7-aec8f7d80e62/volumes" Jul 31 14:51:12.546667 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:12.546580 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" event={"ID":"5f7cb9b3-6905-485d-b735-1d6d5a83d234","Type":"ContainerStarted","Data":"2d86b7a46b9ac06ab659827cca311b38758602febfbe53b313236b8a06909228"} Jul 31 14:51:12.546667 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:12.546640 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" event={"ID":"5f7cb9b3-6905-485d-b735-1d6d5a83d234","Type":"ContainerStarted","Data":"ff8542c110f73da9454fe1a536896bc834ad2aa944824f7e3473b162d74faf3f"} Jul 31 14:51:12.563114 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:12.562993 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" podStartSLOduration=1.5629750009999999 podStartE2EDuration="1.562975001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:51:12.562374673 +0000 UTC m=+2324.599197740" watchObservedRunningTime="2026-07-31 14:51:12.562975001 +0000 UTC m=+2324.599798072" Jul 31 14:51:12.564763 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:12.564728 2577 patch_prober.go:28] interesting pod/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 14:51:12.564889 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:12.564769 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" podUID="3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 14:51:12.907524 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:12.907489 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:13.912159 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:13.912127 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:14.556060 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:14.556033 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:14.556947 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:14.556925 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:15.697193 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:15.697156 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss"] Jul 31 14:51:15.701412 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:15.701396 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss" Jul 31 14:51:15.703170 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:15.703146 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3187acc6\"" Jul 31 14:51:15.703311 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:15.703294 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3187acc6-dockercfg-6v9r2\"" Jul 31 14:51:15.710816 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:15.710792 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss"] Jul 31 14:51:15.737019 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:15.736994 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3cbd22-83c1-4a8a-9233-011378d6d8af-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss\" (UID: \"2e3cbd22-83c1-4a8a-9233-011378d6d8af\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss" Jul 31 14:51:15.737140 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:15.737032 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2wm5\" (UniqueName: \"kubernetes.io/projected/2e3cbd22-83c1-4a8a-9233-011378d6d8af-kube-api-access-l2wm5\") pod \"payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss\" (UID: \"2e3cbd22-83c1-4a8a-9233-011378d6d8af\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss" Jul 31 14:51:15.838315 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:15.838287 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3cbd22-83c1-4a8a-9233-011378d6d8af-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss\" (UID: \"2e3cbd22-83c1-4a8a-9233-011378d6d8af\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss" Jul 31 14:51:15.838419 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:15.838332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2wm5\" (UniqueName: \"kubernetes.io/projected/2e3cbd22-83c1-4a8a-9233-011378d6d8af-kube-api-access-l2wm5\") pod \"payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss\" (UID: \"2e3cbd22-83c1-4a8a-9233-011378d6d8af\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss" Jul 31 14:51:15.839205 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:15.839181 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3cbd22-83c1-4a8a-9233-011378d6d8af-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss\" (UID: \"2e3cbd22-83c1-4a8a-9233-011378d6d8af\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss" Jul 31 14:51:15.845546 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:15.845523 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2wm5\" (UniqueName: \"kubernetes.io/projected/2e3cbd22-83c1-4a8a-9233-011378d6d8af-kube-api-access-l2wm5\") pod \"payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss\" (UID: \"2e3cbd22-83c1-4a8a-9233-011378d6d8af\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss" Jul 31 14:51:16.013149 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.013123 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss" Jul 31 14:51:16.040691 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.040669 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:51:16.140937 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.140902 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-credential-socket\") pod \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " Jul 31 14:51:16.141094 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.140985 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-token\") pod \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " Jul 31 14:51:16.141094 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.141041 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-workload-socket\") pod \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " Jul 31 14:51:16.141208 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.141140 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbjlx\" (UniqueName: \"kubernetes.io/projected/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-kube-api-access-lbjlx\") pod \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " Jul 31 14:51:16.141208 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.141149 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" (UID: "3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:51:16.141291 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.141213 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istiod-ca-cert\") pod \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " Jul 31 14:51:16.141291 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.141229 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" (UID: "3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:51:16.141291 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.141246 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-envoy\") pod \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " Jul 31 14:51:16.141397 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.141290 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-podinfo\") pod \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " Jul 31 14:51:16.141397 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.141340 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-workload-certs\") pod \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " Jul 31 14:51:16.141397 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.141375 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-data\") pod \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\" (UID: \"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c\") " Jul 31 14:51:16.141547 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.141524 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" (UID: "3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:51:16.141756 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.141732 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" (UID: "3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:51:16.141756 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.141748 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" (UID: "3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:51:16.141958 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.141792 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-credential-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:16.141958 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.141813 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-workload-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:16.141958 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.141827 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istiod-ca-cert\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:16.143209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.143186 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" (UID: "3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:51:16.143209 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.143199 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" (UID: "3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:51:16.143668 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.143649 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-kube-api-access-lbjlx" (OuterVolumeSpecName: "kube-api-access-lbjlx") pod "3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" (UID: "3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c"). InnerVolumeSpecName "kube-api-access-lbjlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:51:16.143733 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.143650 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" (UID: "3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 14:51:16.242527 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.242467 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-envoy\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:16.242527 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.242488 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-podinfo\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:16.242527 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.242497 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-workload-certs\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:16.242527 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.242506 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-data\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:16.242527 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.242514 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-istio-token\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:16.242527 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.242522 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbjlx\" (UniqueName: \"kubernetes.io/projected/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c-kube-api-access-lbjlx\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:16.347558 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.347527 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss"] Jul 31 14:51:16.350184 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:51:16.350163 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3cbd22_83c1_4a8a_9233_011378d6d8af.slice/crio-c647b30e98f4bf2d9488e9beb727736b5751b3b14d16cf34c3592ebb05aeb075 WatchSource:0}: Error finding container c647b30e98f4bf2d9488e9beb727736b5751b3b14d16cf34c3592ebb05aeb075: Status 404 returned error can't find the container with id c647b30e98f4bf2d9488e9beb727736b5751b3b14d16cf34c3592ebb05aeb075 Jul 31 14:51:16.567534 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.567455 2577 generic.go:356] "Generic (PLEG): container finished" podID="3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" containerID="9c8cafd3b39a82414cd657cbd318a58682fe6dd0bb280df44cc62431024e4e09" exitCode=0 Jul 31 14:51:16.567534 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.567515 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" Jul 31 14:51:16.567748 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.567543 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" event={"ID":"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c","Type":"ContainerDied","Data":"9c8cafd3b39a82414cd657cbd318a58682fe6dd0bb280df44cc62431024e4e09"} Jul 31 14:51:16.567748 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.567601 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25" event={"ID":"3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c","Type":"ContainerDied","Data":"9638bb5dab701d006f93bf101e90f85dc6fe2d825d07675dc412ab32358ef77c"} Jul 31 14:51:16.567748 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.567630 2577 scope.go:117] "RemoveContainer" containerID="9c8cafd3b39a82414cd657cbd318a58682fe6dd0bb280df44cc62431024e4e09" Jul 31 14:51:16.569253 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.569232 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss" event={"ID":"2e3cbd22-83c1-4a8a-9233-011378d6d8af","Type":"ContainerStarted","Data":"13ea51962cb55e1c8a2c322f7f6a545db5c481b9fddda33373f42a08f1ace816"} Jul 31 14:51:16.569598 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.569269 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss" event={"ID":"2e3cbd22-83c1-4a8a-9233-011378d6d8af","Type":"ContainerStarted","Data":"c647b30e98f4bf2d9488e9beb727736b5751b3b14d16cf34c3592ebb05aeb075"} Jul 31 14:51:16.576049 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.576033 2577 scope.go:117] "RemoveContainer" containerID="6c4bb8f3589c13e076a0ed53f96e6009c443110942c4e5d625a2dc91d0ba814a" Jul 31 14:51:16.581746 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.581723 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25"] Jul 31 14:51:16.583721 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.583707 2577 scope.go:117] "RemoveContainer" containerID="9c8cafd3b39a82414cd657cbd318a58682fe6dd0bb280df44cc62431024e4e09" Jul 31 14:51:16.584023 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:51:16.583999 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c8cafd3b39a82414cd657cbd318a58682fe6dd0bb280df44cc62431024e4e09\": container with ID starting with 9c8cafd3b39a82414cd657cbd318a58682fe6dd0bb280df44cc62431024e4e09 not found: ID does not exist" containerID="9c8cafd3b39a82414cd657cbd318a58682fe6dd0bb280df44cc62431024e4e09" Jul 31 14:51:16.584093 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.584025 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8cafd3b39a82414cd657cbd318a58682fe6dd0bb280df44cc62431024e4e09"} err="failed to get container status \"9c8cafd3b39a82414cd657cbd318a58682fe6dd0bb280df44cc62431024e4e09\": rpc error: code = NotFound desc = could not find container \"9c8cafd3b39a82414cd657cbd318a58682fe6dd0bb280df44cc62431024e4e09\": container with ID starting with 9c8cafd3b39a82414cd657cbd318a58682fe6dd0bb280df44cc62431024e4e09 not found: ID does not exist" Jul 31 14:51:16.584093 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.584047 2577 scope.go:117] "RemoveContainer" containerID="6c4bb8f3589c13e076a0ed53f96e6009c443110942c4e5d625a2dc91d0ba814a" Jul 31 14:51:16.584266 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:51:16.584247 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c4bb8f3589c13e076a0ed53f96e6009c443110942c4e5d625a2dc91d0ba814a\": container with ID starting with 6c4bb8f3589c13e076a0ed53f96e6009c443110942c4e5d625a2dc91d0ba814a not found: ID does not exist" containerID="6c4bb8f3589c13e076a0ed53f96e6009c443110942c4e5d625a2dc91d0ba814a" Jul 31 14:51:16.584321 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.584275 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4bb8f3589c13e076a0ed53f96e6009c443110942c4e5d625a2dc91d0ba814a"} err="failed to get container status \"6c4bb8f3589c13e076a0ed53f96e6009c443110942c4e5d625a2dc91d0ba814a\": rpc error: code = NotFound desc = could not find container \"6c4bb8f3589c13e076a0ed53f96e6009c443110942c4e5d625a2dc91d0ba814a\": container with ID starting with 6c4bb8f3589c13e076a0ed53f96e6009c443110942c4e5d625a2dc91d0ba814a not found: ID does not exist" Jul 31 14:51:16.586898 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.586830 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6c458921-openshift-default-7d7d5fc796-n5r25"] Jul 31 14:51:16.598399 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:16.598359 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss" podStartSLOduration=1.598346112 podStartE2EDuration="1.598346112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:51:16.597011103 +0000 UTC m=+2328.633834170" watchObservedRunningTime="2026-07-31 14:51:16.598346112 +0000 UTC m=+2328.635169178" Jul 31 14:51:17.575073 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:17.575044 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss" Jul 31 14:51:18.168481 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:18.168449 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:51:18.430032 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:18.429950 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" path="/var/lib/kubelet/pods/3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c/volumes" Jul 31 14:51:28.581548 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:28.581522 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss" Jul 31 14:51:28.631368 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:28.631330 2577 generic.go:356] "Generic (PLEG): container finished" podID="5f7cb9b3-6905-485d-b735-1d6d5a83d234" containerID="2d86b7a46b9ac06ab659827cca311b38758602febfbe53b313236b8a06909228" exitCode=0 Jul 31 14:51:28.631506 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:28.631397 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" event={"ID":"5f7cb9b3-6905-485d-b735-1d6d5a83d234","Type":"ContainerDied","Data":"2d86b7a46b9ac06ab659827cca311b38758602febfbe53b313236b8a06909228"} Jul 31 14:51:28.631792 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:28.631778 2577 scope.go:117] "RemoveContainer" containerID="2d86b7a46b9ac06ab659827cca311b38758602febfbe53b313236b8a06909228" Jul 31 14:51:28.632854 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:28.632810 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:51:28.632939 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:28.632919 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:51:28.632980 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:28.632967 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:51:29.556670 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:29.556620 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:29.641452 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:29.641417 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" event={"ID":"5f7cb9b3-6905-485d-b735-1d6d5a83d234","Type":"ContainerStarted","Data":"9e6c8613f8a30dde893f8f931e30bd0b5f0ec9efd207e960aa8cdd4293319f04"} Jul 31 14:51:29.907382 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:29.907347 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:29.908430 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:29.908405 2577 patch_prober.go:28] interesting pod/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 31 14:51:29.908559 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:29.908452 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" podUID="5f7cb9b3-6905-485d-b735-1d6d5a83d234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 31 14:51:30.907726 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:30.907691 2577 patch_prober.go:28] interesting pod/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 31 14:51:30.908162 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:30.907742 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" podUID="5f7cb9b3-6905-485d-b735-1d6d5a83d234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 31 14:51:31.907920 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:31.907878 2577 patch_prober.go:28] interesting pod/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 31 14:51:31.908389 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:31.907945 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" podUID="5f7cb9b3-6905-485d-b735-1d6d5a83d234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 31 14:51:32.907725 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:32.907680 2577 patch_prober.go:28] interesting pod/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 31 14:51:32.907946 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:32.907743 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" podUID="5f7cb9b3-6905-485d-b735-1d6d5a83d234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 31 14:51:33.908257 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:33.908219 2577 patch_prober.go:28] interesting pod/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 31 14:51:33.908665 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:33.908289 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" podUID="5f7cb9b3-6905-485d-b735-1d6d5a83d234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 31 14:51:34.911610 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:34.911576 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:34.912062 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:34.911861 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:34.912562 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:34.912543 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:45.550322 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:45.550281 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss"] Jul 31 14:51:45.550723 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:45.550574 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss" podUID="2e3cbd22-83c1-4a8a-9233-011378d6d8af" containerName="payload-pre-processing" containerID="cri-o://13ea51962cb55e1c8a2c322f7f6a545db5c481b9fddda33373f42a08f1ace816" gracePeriod=30 Jul 31 14:51:45.715149 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:45.715120 2577 generic.go:356] "Generic (PLEG): container finished" podID="2e3cbd22-83c1-4a8a-9233-011378d6d8af" containerID="13ea51962cb55e1c8a2c322f7f6a545db5c481b9fddda33373f42a08f1ace816" exitCode=0 Jul 31 14:51:45.715306 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:45.715201 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss" event={"ID":"2e3cbd22-83c1-4a8a-9233-011378d6d8af","Type":"ContainerDied","Data":"13ea51962cb55e1c8a2c322f7f6a545db5c481b9fddda33373f42a08f1ace816"} Jul 31 14:51:45.803607 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:45.803581 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl"] Jul 31 14:51:45.803986 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:45.803831 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" podUID="5f7cb9b3-6905-485d-b735-1d6d5a83d234" containerName="istio-proxy" containerID="cri-o://9e6c8613f8a30dde893f8f931e30bd0b5f0ec9efd207e960aa8cdd4293319f04" gracePeriod=30 Jul 31 14:51:45.811155 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:45.811133 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss" Jul 31 14:51:45.885312 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:45.885280 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2wm5\" (UniqueName: \"kubernetes.io/projected/2e3cbd22-83c1-4a8a-9233-011378d6d8af-kube-api-access-l2wm5\") pod \"2e3cbd22-83c1-4a8a-9233-011378d6d8af\" (UID: \"2e3cbd22-83c1-4a8a-9233-011378d6d8af\") " Jul 31 14:51:45.885312 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:45.885331 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3cbd22-83c1-4a8a-9233-011378d6d8af-plugins-config-volume\") pod \"2e3cbd22-83c1-4a8a-9233-011378d6d8af\" (UID: \"2e3cbd22-83c1-4a8a-9233-011378d6d8af\") " Jul 31 14:51:45.885751 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:45.885723 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3cbd22-83c1-4a8a-9233-011378d6d8af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2e3cbd22-83c1-4a8a-9233-011378d6d8af" (UID: "2e3cbd22-83c1-4a8a-9233-011378d6d8af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:51:45.887315 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:45.887291 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3cbd22-83c1-4a8a-9233-011378d6d8af-kube-api-access-l2wm5" (OuterVolumeSpecName: "kube-api-access-l2wm5") pod "2e3cbd22-83c1-4a8a-9233-011378d6d8af" (UID: "2e3cbd22-83c1-4a8a-9233-011378d6d8af"). InnerVolumeSpecName "kube-api-access-l2wm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:51:45.987299 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:45.987265 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2wm5\" (UniqueName: \"kubernetes.io/projected/2e3cbd22-83c1-4a8a-9233-011378d6d8af-kube-api-access-l2wm5\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:45.987484 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:45.987299 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3cbd22-83c1-4a8a-9233-011378d6d8af-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:46.720873 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:46.720771 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss" event={"ID":"2e3cbd22-83c1-4a8a-9233-011378d6d8af","Type":"ContainerDied","Data":"c647b30e98f4bf2d9488e9beb727736b5751b3b14d16cf34c3592ebb05aeb075"} Jul 31 14:51:46.720873 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:46.720799 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss" Jul 31 14:51:46.720873 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:46.720832 2577 scope.go:117] "RemoveContainer" containerID="13ea51962cb55e1c8a2c322f7f6a545db5c481b9fddda33373f42a08f1ace816" Jul 31 14:51:46.736549 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:46.736528 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss"] Jul 31 14:51:46.742234 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:46.742209 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3187acc6-5f86f88588-vhlss"] Jul 31 14:51:48.429326 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:48.429293 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e3cbd22-83c1-4a8a-9233-011378d6d8af" path="/var/lib/kubelet/pods/2e3cbd22-83c1-4a8a-9233-011378d6d8af/volumes" Jul 31 14:51:48.479648 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:48.479618 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:51:49.914247 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:49.914206 2577 patch_prober.go:28] interesting pod/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 14:51:49.914607 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:49.914274 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" podUID="5f7cb9b3-6905-485d-b735-1d6d5a83d234" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 14:51:51.046124 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.046097 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:51.136039 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.136008 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-data\") pod \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " Jul 31 14:51:51.136174 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.136057 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-envoy\") pod \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " Jul 31 14:51:51.136174 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.136081 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-credential-socket\") pod \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " Jul 31 14:51:51.136174 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.136111 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istiod-ca-cert\") pod \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " Jul 31 14:51:51.136174 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.136145 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-podinfo\") pod \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " Jul 31 14:51:51.136389 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.136183 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-workload-certs\") pod \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " Jul 31 14:51:51.136389 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.136206 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-workload-socket\") pod \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " Jul 31 14:51:51.136389 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.136242 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffk7t\" (UniqueName: \"kubernetes.io/projected/5f7cb9b3-6905-485d-b735-1d6d5a83d234-kube-api-access-ffk7t\") pod \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " Jul 31 14:51:51.136389 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.136266 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-token\") pod \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\" (UID: \"5f7cb9b3-6905-485d-b735-1d6d5a83d234\") " Jul 31 14:51:51.136389 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.136345 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-data" (OuterVolumeSpecName: "istio-data") pod "5f7cb9b3-6905-485d-b735-1d6d5a83d234" (UID: "5f7cb9b3-6905-485d-b735-1d6d5a83d234"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:51:51.136637 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.136433 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5f7cb9b3-6905-485d-b735-1d6d5a83d234" (UID: "5f7cb9b3-6905-485d-b735-1d6d5a83d234"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:51:51.136637 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.136524 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5f7cb9b3-6905-485d-b735-1d6d5a83d234" (UID: "5f7cb9b3-6905-485d-b735-1d6d5a83d234"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:51:51.136637 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.136612 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5f7cb9b3-6905-485d-b735-1d6d5a83d234" (UID: "5f7cb9b3-6905-485d-b735-1d6d5a83d234"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:51:51.136637 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.136632 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-workload-certs\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:51.136861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.136652 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-data\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:51.136861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.136667 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-credential-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:51.136861 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.136809 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5f7cb9b3-6905-485d-b735-1d6d5a83d234" (UID: "5f7cb9b3-6905-485d-b735-1d6d5a83d234"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:51:51.138373 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.138348 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-token" (OuterVolumeSpecName: "istio-token") pod "5f7cb9b3-6905-485d-b735-1d6d5a83d234" (UID: "5f7cb9b3-6905-485d-b735-1d6d5a83d234"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:51:51.138481 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.138432 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7cb9b3-6905-485d-b735-1d6d5a83d234-kube-api-access-ffk7t" (OuterVolumeSpecName: "kube-api-access-ffk7t") pod "5f7cb9b3-6905-485d-b735-1d6d5a83d234" (UID: "5f7cb9b3-6905-485d-b735-1d6d5a83d234"). InnerVolumeSpecName "kube-api-access-ffk7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:51:51.138551 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.138536 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5f7cb9b3-6905-485d-b735-1d6d5a83d234" (UID: "5f7cb9b3-6905-485d-b735-1d6d5a83d234"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:51:51.138610 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.138569 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5f7cb9b3-6905-485d-b735-1d6d5a83d234" (UID: "5f7cb9b3-6905-485d-b735-1d6d5a83d234"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 14:51:51.238104 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.238068 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-workload-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:51.238104 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.238099 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffk7t\" (UniqueName: \"kubernetes.io/projected/5f7cb9b3-6905-485d-b735-1d6d5a83d234-kube-api-access-ffk7t\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:51.238104 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.238111 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-token\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:51.238316 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.238122 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-envoy\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:51.238316 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.238134 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istiod-ca-cert\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:51.238316 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.238144 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7cb9b3-6905-485d-b735-1d6d5a83d234-istio-podinfo\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:51:51.743858 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.743820 2577 generic.go:356] "Generic (PLEG): container finished" podID="5f7cb9b3-6905-485d-b735-1d6d5a83d234" containerID="9e6c8613f8a30dde893f8f931e30bd0b5f0ec9efd207e960aa8cdd4293319f04" exitCode=0 Jul 31 14:51:51.744041 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.743882 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" event={"ID":"5f7cb9b3-6905-485d-b735-1d6d5a83d234","Type":"ContainerDied","Data":"9e6c8613f8a30dde893f8f931e30bd0b5f0ec9efd207e960aa8cdd4293319f04"} Jul 31 14:51:51.744041 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.743921 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" event={"ID":"5f7cb9b3-6905-485d-b735-1d6d5a83d234","Type":"ContainerDied","Data":"ff8542c110f73da9454fe1a536896bc834ad2aa944824f7e3473b162d74faf3f"} Jul 31 14:51:51.744041 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.743943 2577 scope.go:117] "RemoveContainer" containerID="9e6c8613f8a30dde893f8f931e30bd0b5f0ec9efd207e960aa8cdd4293319f04" Jul 31 14:51:51.744041 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.743884 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl" Jul 31 14:51:51.752369 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.752353 2577 scope.go:117] "RemoveContainer" containerID="2d86b7a46b9ac06ab659827cca311b38758602febfbe53b313236b8a06909228" Jul 31 14:51:51.759189 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.759165 2577 scope.go:117] "RemoveContainer" containerID="9e6c8613f8a30dde893f8f931e30bd0b5f0ec9efd207e960aa8cdd4293319f04" Jul 31 14:51:51.759459 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:51:51.759441 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6c8613f8a30dde893f8f931e30bd0b5f0ec9efd207e960aa8cdd4293319f04\": container with ID starting with 9e6c8613f8a30dde893f8f931e30bd0b5f0ec9efd207e960aa8cdd4293319f04 not found: ID does not exist" containerID="9e6c8613f8a30dde893f8f931e30bd0b5f0ec9efd207e960aa8cdd4293319f04" Jul 31 14:51:51.759524 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.759467 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6c8613f8a30dde893f8f931e30bd0b5f0ec9efd207e960aa8cdd4293319f04"} err="failed to get container status \"9e6c8613f8a30dde893f8f931e30bd0b5f0ec9efd207e960aa8cdd4293319f04\": rpc error: code = NotFound desc = could not find container \"9e6c8613f8a30dde893f8f931e30bd0b5f0ec9efd207e960aa8cdd4293319f04\": container with ID starting with 9e6c8613f8a30dde893f8f931e30bd0b5f0ec9efd207e960aa8cdd4293319f04 not found: ID does not exist" Jul 31 14:51:51.759524 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.759486 2577 scope.go:117] "RemoveContainer" containerID="2d86b7a46b9ac06ab659827cca311b38758602febfbe53b313236b8a06909228" Jul 31 14:51:51.759706 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:51:51.759685 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d86b7a46b9ac06ab659827cca311b38758602febfbe53b313236b8a06909228\": container with ID starting with 2d86b7a46b9ac06ab659827cca311b38758602febfbe53b313236b8a06909228 not found: ID does not exist" containerID="2d86b7a46b9ac06ab659827cca311b38758602febfbe53b313236b8a06909228" Jul 31 14:51:51.759757 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.759715 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d86b7a46b9ac06ab659827cca311b38758602febfbe53b313236b8a06909228"} err="failed to get container status \"2d86b7a46b9ac06ab659827cca311b38758602febfbe53b313236b8a06909228\": rpc error: code = NotFound desc = could not find container \"2d86b7a46b9ac06ab659827cca311b38758602febfbe53b313236b8a06909228\": container with ID starting with 2d86b7a46b9ac06ab659827cca311b38758602febfbe53b313236b8a06909228 not found: ID does not exist" Jul 31 14:51:51.763154 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.763132 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl"] Jul 31 14:51:51.766925 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:51.766907 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3187acc6-openshift-default-6fbd9ff776-8phgl"] Jul 31 14:51:52.428952 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:52.428916 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f7cb9b3-6905-485d-b735-1d6d5a83d234" path="/var/lib/kubelet/pods/5f7cb9b3-6905-485d-b735-1d6d5a83d234/volumes" Jul 31 14:51:59.979658 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:51:59.979616 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:53:31.078544 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:31.078512 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:53:39.167415 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:39.167384 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:53:43.217101 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.217069 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd"] Jul 31 14:53:43.217503 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.217397 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f7cb9b3-6905-485d-b735-1d6d5a83d234" containerName="istio-proxy" Jul 31 14:53:43.217503 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.217409 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f7cb9b3-6905-485d-b735-1d6d5a83d234" containerName="istio-proxy" Jul 31 14:53:43.217503 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.217419 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e3cbd22-83c1-4a8a-9233-011378d6d8af" containerName="payload-pre-processing" Jul 31 14:53:43.217503 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.217425 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e3cbd22-83c1-4a8a-9233-011378d6d8af" containerName="payload-pre-processing" Jul 31 14:53:43.217503 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.217440 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f7cb9b3-6905-485d-b735-1d6d5a83d234" containerName="istio-proxy" Jul 31 14:53:43.217503 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.217445 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f7cb9b3-6905-485d-b735-1d6d5a83d234" containerName="istio-proxy" Jul 31 14:53:43.217503 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.217456 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" containerName="istio-proxy" Jul 31 14:53:43.217503 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.217461 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" containerName="istio-proxy" Jul 31 14:53:43.217503 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.217466 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" containerName="istio-proxy" Jul 31 14:53:43.217503 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.217471 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" containerName="istio-proxy" Jul 31 14:53:43.217825 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.217517 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f7cb9b3-6905-485d-b735-1d6d5a83d234" containerName="istio-proxy" Jul 31 14:53:43.217825 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.217526 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" containerName="istio-proxy" Jul 31 14:53:43.217825 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.217532 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e3cbd22-83c1-4a8a-9233-011378d6d8af" containerName="payload-pre-processing" Jul 31 14:53:43.220393 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.220378 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.222456 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.222434 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7f0814aa-openshift-default-dockercfg-dxjfd\"" Jul 31 14:53:43.238994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.238972 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd"] Jul 31 14:53:43.359734 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.359695 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-credential-socket\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.359911 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.359750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-workload-certs\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.359911 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.359778 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-podinfo\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.359911 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.359815 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istiod-ca-cert\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.359911 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.359869 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svxng\" (UniqueName: \"kubernetes.io/projected/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-kube-api-access-svxng\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.359911 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.359896 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-envoy\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.360082 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.359927 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-workload-socket\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.360082 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.359958 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-token\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.360082 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.360008 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-data\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.460949 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.460919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-credential-socket\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.461127 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.460973 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-workload-certs\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.461127 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.460999 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-podinfo\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.461127 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.461023 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istiod-ca-cert\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.461127 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.461067 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svxng\" (UniqueName: \"kubernetes.io/projected/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-kube-api-access-svxng\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.461127 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.461107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-envoy\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.461402 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.461148 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-workload-socket\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.461402 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.461178 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-token\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.461402 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.461212 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-data\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.461788 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.461757 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-credential-socket\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.462059 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.461810 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-workload-certs\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.462142 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.461925 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-workload-socket\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.462142 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.462093 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-data\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.464263 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.464238 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-envoy\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.464470 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.464445 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-podinfo\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.464559 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.464432 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istiod-ca-cert\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.469823 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.469773 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-token\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.470331 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.470315 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svxng\" (UniqueName: \"kubernetes.io/projected/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-kube-api-access-svxng\") pod \"e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.530241 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.530214 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:43.664048 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.664024 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd"] Jul 31 14:53:43.666614 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:53:43.666584 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb56dcb2_ea04_4a31_a5c6_79a38d24a834.slice/crio-8e15f6243e639f64553ba80b6f647c490550f2b753949eef149b50efa1e6a0c6 WatchSource:0}: Error finding container 8e15f6243e639f64553ba80b6f647c490550f2b753949eef149b50efa1e6a0c6: Status 404 returned error can't find the container with id 8e15f6243e639f64553ba80b6f647c490550f2b753949eef149b50efa1e6a0c6 Jul 31 14:53:43.668694 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.668662 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:53:43.668760 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.668726 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:53:43.668760 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:43.668755 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:53:44.216203 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:44.216167 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" event={"ID":"fb56dcb2-ea04-4a31-a5c6-79a38d24a834","Type":"ContainerStarted","Data":"cc60301077af993d040cdebad498911f9ec906b141e6f630eca4a96fcfd189ac"} Jul 31 14:53:44.216414 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:44.216228 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" event={"ID":"fb56dcb2-ea04-4a31-a5c6-79a38d24a834","Type":"ContainerStarted","Data":"8e15f6243e639f64553ba80b6f647c490550f2b753949eef149b50efa1e6a0c6"} Jul 31 14:53:44.235951 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:44.235898 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" podStartSLOduration=1.235883771 podStartE2EDuration="1.235883771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:53:44.234414369 +0000 UTC m=+2476.271237436" watchObservedRunningTime="2026-07-31 14:53:44.235883771 +0000 UTC m=+2476.272706838" Jul 31 14:53:44.530825 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:44.530742 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:44.535813 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:44.535783 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:45.220411 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:45.220378 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:45.221409 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:45.221393 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:53:47.841034 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:47.840995 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr"] Jul 31 14:53:47.841412 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:47.841349 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f7cb9b3-6905-485d-b735-1d6d5a83d234" containerName="istio-proxy" Jul 31 14:53:47.841412 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:47.841364 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ec56faf-cd5c-4cc3-b02c-b09fadae0e9c" containerName="istio-proxy" Jul 31 14:53:47.844313 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:47.844297 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr" Jul 31 14:53:47.846336 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:47.846317 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7f0814aa\"" Jul 31 14:53:47.853530 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:47.853507 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr"] Jul 31 14:53:47.879743 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:47.879716 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7"] Jul 31 14:53:47.882866 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:47.882833 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7" Jul 31 14:53:47.885208 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:47.885186 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7f0814aa-dockercfg-zlfk7\"" Jul 31 14:53:47.895523 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:47.895501 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7"] Jul 31 14:53:48.001269 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:48.001238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad959dc-ed8c-4764-ac5b-9ea331a5a906-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr\" (UID: \"7ad959dc-ed8c-4764-ac5b-9ea331a5a906\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr" Jul 31 14:53:48.001269 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:48.001278 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4214b7-bfcf-4c83-ab47-c727e1a8a483-plugins-config-volume\") pod \"payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7\" (UID: \"7c4214b7-bfcf-4c83-ab47-c727e1a8a483\") " pod="openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7" Jul 31 14:53:48.001479 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:48.001301 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr4mq\" (UniqueName: \"kubernetes.io/projected/7c4214b7-bfcf-4c83-ab47-c727e1a8a483-kube-api-access-pr4mq\") pod \"payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7\" (UID: \"7c4214b7-bfcf-4c83-ab47-c727e1a8a483\") " pod="openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7" Jul 31 14:53:48.001479 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:48.001416 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwjwf\" (UniqueName: \"kubernetes.io/projected/7ad959dc-ed8c-4764-ac5b-9ea331a5a906-kube-api-access-qwjwf\") pod \"payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr\" (UID: \"7ad959dc-ed8c-4764-ac5b-9ea331a5a906\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr" Jul 31 14:53:48.102693 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:48.102618 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad959dc-ed8c-4764-ac5b-9ea331a5a906-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr\" (UID: \"7ad959dc-ed8c-4764-ac5b-9ea331a5a906\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr" Jul 31 14:53:48.102693 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:48.102654 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4214b7-bfcf-4c83-ab47-c727e1a8a483-plugins-config-volume\") pod \"payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7\" (UID: \"7c4214b7-bfcf-4c83-ab47-c727e1a8a483\") " pod="openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7" Jul 31 14:53:48.102693 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:48.102673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr4mq\" (UniqueName: \"kubernetes.io/projected/7c4214b7-bfcf-4c83-ab47-c727e1a8a483-kube-api-access-pr4mq\") pod \"payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7\" (UID: \"7c4214b7-bfcf-4c83-ab47-c727e1a8a483\") " pod="openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7" Jul 31 14:53:48.102930 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:48.102727 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwjwf\" (UniqueName: \"kubernetes.io/projected/7ad959dc-ed8c-4764-ac5b-9ea331a5a906-kube-api-access-qwjwf\") pod \"payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr\" (UID: \"7ad959dc-ed8c-4764-ac5b-9ea331a5a906\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr" Jul 31 14:53:48.103359 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:48.103338 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad959dc-ed8c-4764-ac5b-9ea331a5a906-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr\" (UID: \"7ad959dc-ed8c-4764-ac5b-9ea331a5a906\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr" Jul 31 14:53:48.103448 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:48.103430 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4214b7-bfcf-4c83-ab47-c727e1a8a483-plugins-config-volume\") pod \"payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7\" (UID: \"7c4214b7-bfcf-4c83-ab47-c727e1a8a483\") " pod="openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7" Jul 31 14:53:48.111222 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:48.111203 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwjwf\" (UniqueName: \"kubernetes.io/projected/7ad959dc-ed8c-4764-ac5b-9ea331a5a906-kube-api-access-qwjwf\") pod \"payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr\" (UID: \"7ad959dc-ed8c-4764-ac5b-9ea331a5a906\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr" Jul 31 14:53:48.111330 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:48.111266 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr4mq\" (UniqueName: \"kubernetes.io/projected/7c4214b7-bfcf-4c83-ab47-c727e1a8a483-kube-api-access-pr4mq\") pod \"payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7\" (UID: \"7c4214b7-bfcf-4c83-ab47-c727e1a8a483\") " pod="openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7" Jul 31 14:53:48.155741 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:48.155719 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr" Jul 31 14:53:48.191975 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:48.191948 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7" Jul 31 14:53:48.297238 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:48.297207 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr"] Jul 31 14:53:48.300013 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:53:48.299971 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad959dc_ed8c_4764_ac5b_9ea331a5a906.slice/crio-79fe7e7d03e217e7a5dd14d04a6cd37f5b787cf5badabaa8ab421dfd199b1182 WatchSource:0}: Error finding container 79fe7e7d03e217e7a5dd14d04a6cd37f5b787cf5badabaa8ab421dfd199b1182: Status 404 returned error can't find the container with id 79fe7e7d03e217e7a5dd14d04a6cd37f5b787cf5badabaa8ab421dfd199b1182 Jul 31 14:53:48.330612 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:48.330587 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7"] Jul 31 14:53:48.333124 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:53:48.333093 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4214b7_bfcf_4c83_ab47_c727e1a8a483.slice/crio-b3a8084eb75139b70328527ebf595596dfe107092f89e8733635a775bc7b98ae WatchSource:0}: Error finding container b3a8084eb75139b70328527ebf595596dfe107092f89e8733635a775bc7b98ae: Status 404 returned error can't find the container with id b3a8084eb75139b70328527ebf595596dfe107092f89e8733635a775bc7b98ae Jul 31 14:53:49.240560 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:49.240518 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7" event={"ID":"7c4214b7-bfcf-4c83-ab47-c727e1a8a483","Type":"ContainerStarted","Data":"8b5f225bf59b9b2c6b11068c1b8c249d1f57c89d76dcc1d37078aefe1464225f"} Jul 31 14:53:49.241056 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:49.240584 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7" event={"ID":"7c4214b7-bfcf-4c83-ab47-c727e1a8a483","Type":"ContainerStarted","Data":"b3a8084eb75139b70328527ebf595596dfe107092f89e8733635a775bc7b98ae"} Jul 31 14:53:49.241056 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:49.240628 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7" Jul 31 14:53:49.242107 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:49.242085 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr" event={"ID":"7ad959dc-ed8c-4764-ac5b-9ea331a5a906","Type":"ContainerStarted","Data":"582596a26560ca6959a6a5d43339d5a38910cd678db3c8c162355b8aadc367aa"} Jul 31 14:53:49.242231 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:49.242122 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr" event={"ID":"7ad959dc-ed8c-4764-ac5b-9ea331a5a906","Type":"ContainerStarted","Data":"79fe7e7d03e217e7a5dd14d04a6cd37f5b787cf5badabaa8ab421dfd199b1182"} Jul 31 14:53:49.242231 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:49.242141 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr" Jul 31 14:53:49.256481 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:49.256434 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7" podStartSLOduration=2.256421423 podStartE2EDuration="2.256421423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:53:49.254860864 +0000 UTC m=+2481.291683927" watchObservedRunningTime="2026-07-31 14:53:49.256421423 +0000 UTC m=+2481.293244489" Jul 31 14:53:49.268918 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:49.268879 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr" podStartSLOduration=2.268867538 podStartE2EDuration="2.268867538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:53:49.267724337 +0000 UTC m=+2481.304547405" watchObservedRunningTime="2026-07-31 14:53:49.268867538 +0000 UTC m=+2481.305690599" Jul 31 14:53:49.678775 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:49.678742 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:53:59.289043 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:59.289014 2577 generic.go:356] "Generic (PLEG): container finished" podID="fb56dcb2-ea04-4a31-a5c6-79a38d24a834" containerID="cc60301077af993d040cdebad498911f9ec906b141e6f630eca4a96fcfd189ac" exitCode=0 Jul 31 14:53:59.289415 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:59.289088 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" event={"ID":"fb56dcb2-ea04-4a31-a5c6-79a38d24a834","Type":"ContainerDied","Data":"cc60301077af993d040cdebad498911f9ec906b141e6f630eca4a96fcfd189ac"} Jul 31 14:53:59.289478 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:59.289463 2577 scope.go:117] "RemoveContainer" containerID="cc60301077af993d040cdebad498911f9ec906b141e6f630eca4a96fcfd189ac" Jul 31 14:53:59.290484 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:59.290456 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:53:59.290557 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:59.290539 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:53:59.290601 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:53:59.290585 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:54:00.220391 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:00.220361 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:54:00.249884 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:00.249828 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr" Jul 31 14:54:00.251008 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:00.250987 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7" Jul 31 14:54:00.296731 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:00.296700 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" event={"ID":"fb56dcb2-ea04-4a31-a5c6-79a38d24a834","Type":"ContainerStarted","Data":"5e013bb783c31623fcc85b4564ea2869a0ad134c67691bf976d9f655be3e7d25"} Jul 31 14:54:00.531125 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:00.531029 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:54:00.531528 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:00.531496 2577 patch_prober.go:28] interesting pod/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 31 14:54:00.531622 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:00.531554 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" podUID="fb56dcb2-ea04-4a31-a5c6-79a38d24a834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 31 14:54:01.539444 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:01.539399 2577 patch_prober.go:28] interesting pod/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 31 14:54:01.539995 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:01.539495 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" podUID="fb56dcb2-ea04-4a31-a5c6-79a38d24a834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 31 14:54:02.530777 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:02.530728 2577 patch_prober.go:28] interesting pod/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 31 14:54:02.531019 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:02.530801 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" podUID="fb56dcb2-ea04-4a31-a5c6-79a38d24a834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 31 14:54:03.531326 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:03.531282 2577 patch_prober.go:28] interesting pod/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 31 14:54:03.531859 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:03.531361 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" podUID="fb56dcb2-ea04-4a31-a5c6-79a38d24a834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 31 14:54:04.531463 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:04.531415 2577 patch_prober.go:28] interesting pod/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 31 14:54:04.532002 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:04.531496 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" podUID="fb56dcb2-ea04-4a31-a5c6-79a38d24a834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 31 14:54:05.534628 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:05.534597 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:54:05.535098 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:05.534854 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:54:05.535614 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:05.535597 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:54:28.861498 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:28.861297 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7"] Jul 31 14:54:28.862101 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:28.862068 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7" podUID="7c4214b7-bfcf-4c83-ab47-c727e1a8a483" containerName="payload-processing" containerID="cri-o://8b5f225bf59b9b2c6b11068c1b8c249d1f57c89d76dcc1d37078aefe1464225f" gracePeriod=30 Jul 31 14:54:28.865088 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:28.865059 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr"] Jul 31 14:54:28.865326 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:28.865280 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr" podUID="7ad959dc-ed8c-4764-ac5b-9ea331a5a906" containerName="payload-pre-processing" containerID="cri-o://582596a26560ca6959a6a5d43339d5a38910cd678db3c8c162355b8aadc367aa" gracePeriod=30 Jul 31 14:54:29.120741 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.120673 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7" Jul 31 14:54:29.151478 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.151447 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr" Jul 31 14:54:29.280157 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.280115 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad959dc-ed8c-4764-ac5b-9ea331a5a906-plugins-config-volume\") pod \"7ad959dc-ed8c-4764-ac5b-9ea331a5a906\" (UID: \"7ad959dc-ed8c-4764-ac5b-9ea331a5a906\") " Jul 31 14:54:29.280421 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.280210 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwjwf\" (UniqueName: \"kubernetes.io/projected/7ad959dc-ed8c-4764-ac5b-9ea331a5a906-kube-api-access-qwjwf\") pod \"7ad959dc-ed8c-4764-ac5b-9ea331a5a906\" (UID: \"7ad959dc-ed8c-4764-ac5b-9ea331a5a906\") " Jul 31 14:54:29.280421 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.280285 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4214b7-bfcf-4c83-ab47-c727e1a8a483-plugins-config-volume\") pod \"7c4214b7-bfcf-4c83-ab47-c727e1a8a483\" (UID: \"7c4214b7-bfcf-4c83-ab47-c727e1a8a483\") " Jul 31 14:54:29.280421 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.280317 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pr4mq\" (UniqueName: \"kubernetes.io/projected/7c4214b7-bfcf-4c83-ab47-c727e1a8a483-kube-api-access-pr4mq\") pod \"7c4214b7-bfcf-4c83-ab47-c727e1a8a483\" (UID: \"7c4214b7-bfcf-4c83-ab47-c727e1a8a483\") " Jul 31 14:54:29.280582 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.280479 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad959dc-ed8c-4764-ac5b-9ea331a5a906-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ad959dc-ed8c-4764-ac5b-9ea331a5a906" (UID: "7ad959dc-ed8c-4764-ac5b-9ea331a5a906"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:54:29.280625 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.280581 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad959dc-ed8c-4764-ac5b-9ea331a5a906-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:54:29.280671 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.280604 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c4214b7-bfcf-4c83-ab47-c727e1a8a483-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c4214b7-bfcf-4c83-ab47-c727e1a8a483" (UID: "7c4214b7-bfcf-4c83-ab47-c727e1a8a483"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:54:29.282397 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.282366 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4214b7-bfcf-4c83-ab47-c727e1a8a483-kube-api-access-pr4mq" (OuterVolumeSpecName: "kube-api-access-pr4mq") pod "7c4214b7-bfcf-4c83-ab47-c727e1a8a483" (UID: "7c4214b7-bfcf-4c83-ab47-c727e1a8a483"). InnerVolumeSpecName "kube-api-access-pr4mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:54:29.282397 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.282384 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad959dc-ed8c-4764-ac5b-9ea331a5a906-kube-api-access-qwjwf" (OuterVolumeSpecName: "kube-api-access-qwjwf") pod "7ad959dc-ed8c-4764-ac5b-9ea331a5a906" (UID: "7ad959dc-ed8c-4764-ac5b-9ea331a5a906"). InnerVolumeSpecName "kube-api-access-qwjwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:54:29.381557 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.381524 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwjwf\" (UniqueName: \"kubernetes.io/projected/7ad959dc-ed8c-4764-ac5b-9ea331a5a906-kube-api-access-qwjwf\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:54:29.381557 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.381554 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4214b7-bfcf-4c83-ab47-c727e1a8a483-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:54:29.381727 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.381567 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr4mq\" (UniqueName: \"kubernetes.io/projected/7c4214b7-bfcf-4c83-ab47-c727e1a8a483-kube-api-access-pr4mq\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:54:29.429417 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.429387 2577 generic.go:356] "Generic (PLEG): container finished" podID="7c4214b7-bfcf-4c83-ab47-c727e1a8a483" containerID="8b5f225bf59b9b2c6b11068c1b8c249d1f57c89d76dcc1d37078aefe1464225f" exitCode=0 Jul 31 14:54:29.429548 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.429419 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7" Jul 31 14:54:29.429548 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.429462 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7" event={"ID":"7c4214b7-bfcf-4c83-ab47-c727e1a8a483","Type":"ContainerDied","Data":"8b5f225bf59b9b2c6b11068c1b8c249d1f57c89d76dcc1d37078aefe1464225f"} Jul 31 14:54:29.429548 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.429500 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7" event={"ID":"7c4214b7-bfcf-4c83-ab47-c727e1a8a483","Type":"ContainerDied","Data":"b3a8084eb75139b70328527ebf595596dfe107092f89e8733635a775bc7b98ae"} Jul 31 14:54:29.429548 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.429524 2577 scope.go:117] "RemoveContainer" containerID="8b5f225bf59b9b2c6b11068c1b8c249d1f57c89d76dcc1d37078aefe1464225f" Jul 31 14:54:29.430709 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.430684 2577 generic.go:356] "Generic (PLEG): container finished" podID="7ad959dc-ed8c-4764-ac5b-9ea331a5a906" containerID="582596a26560ca6959a6a5d43339d5a38910cd678db3c8c162355b8aadc367aa" exitCode=0 Jul 31 14:54:29.430813 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.430714 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr" event={"ID":"7ad959dc-ed8c-4764-ac5b-9ea331a5a906","Type":"ContainerDied","Data":"582596a26560ca6959a6a5d43339d5a38910cd678db3c8c162355b8aadc367aa"} Jul 31 14:54:29.430813 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.430734 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr" Jul 31 14:54:29.430813 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.430747 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr" event={"ID":"7ad959dc-ed8c-4764-ac5b-9ea331a5a906","Type":"ContainerDied","Data":"79fe7e7d03e217e7a5dd14d04a6cd37f5b787cf5badabaa8ab421dfd199b1182"} Jul 31 14:54:29.439057 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.439042 2577 scope.go:117] "RemoveContainer" containerID="8b5f225bf59b9b2c6b11068c1b8c249d1f57c89d76dcc1d37078aefe1464225f" Jul 31 14:54:29.439373 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:54:29.439309 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5f225bf59b9b2c6b11068c1b8c249d1f57c89d76dcc1d37078aefe1464225f\": container with ID starting with 8b5f225bf59b9b2c6b11068c1b8c249d1f57c89d76dcc1d37078aefe1464225f not found: ID does not exist" containerID="8b5f225bf59b9b2c6b11068c1b8c249d1f57c89d76dcc1d37078aefe1464225f" Jul 31 14:54:29.439373 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.439343 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5f225bf59b9b2c6b11068c1b8c249d1f57c89d76dcc1d37078aefe1464225f"} err="failed to get container status \"8b5f225bf59b9b2c6b11068c1b8c249d1f57c89d76dcc1d37078aefe1464225f\": rpc error: code = NotFound desc = could not find container \"8b5f225bf59b9b2c6b11068c1b8c249d1f57c89d76dcc1d37078aefe1464225f\": container with ID starting with 8b5f225bf59b9b2c6b11068c1b8c249d1f57c89d76dcc1d37078aefe1464225f not found: ID does not exist" Jul 31 14:54:29.439373 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.439367 2577 scope.go:117] "RemoveContainer" containerID="582596a26560ca6959a6a5d43339d5a38910cd678db3c8c162355b8aadc367aa" Jul 31 14:54:29.446794 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.446779 2577 scope.go:117] "RemoveContainer" containerID="582596a26560ca6959a6a5d43339d5a38910cd678db3c8c162355b8aadc367aa" Jul 31 14:54:29.447061 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:54:29.447042 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582596a26560ca6959a6a5d43339d5a38910cd678db3c8c162355b8aadc367aa\": container with ID starting with 582596a26560ca6959a6a5d43339d5a38910cd678db3c8c162355b8aadc367aa not found: ID does not exist" containerID="582596a26560ca6959a6a5d43339d5a38910cd678db3c8c162355b8aadc367aa" Jul 31 14:54:29.447119 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.447067 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582596a26560ca6959a6a5d43339d5a38910cd678db3c8c162355b8aadc367aa"} err="failed to get container status \"582596a26560ca6959a6a5d43339d5a38910cd678db3c8c162355b8aadc367aa\": rpc error: code = NotFound desc = could not find container \"582596a26560ca6959a6a5d43339d5a38910cd678db3c8c162355b8aadc367aa\": container with ID starting with 582596a26560ca6959a6a5d43339d5a38910cd678db3c8c162355b8aadc367aa not found: ID does not exist" Jul 31 14:54:29.451608 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.451588 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7"] Jul 31 14:54:29.456975 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.456954 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7f0814aa-df6cd5db6-f7cq7"] Jul 31 14:54:29.465265 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.465243 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr"] Jul 31 14:54:29.470904 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:29.470877 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f0814aa-6f56974cbd-sw4vr"] Jul 31 14:54:30.429635 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:30.429605 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ad959dc-ed8c-4764-ac5b-9ea331a5a906" path="/var/lib/kubelet/pods/7ad959dc-ed8c-4764-ac5b-9ea331a5a906/volumes" Jul 31 14:54:30.430027 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:30.429957 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c4214b7-bfcf-4c83-ab47-c727e1a8a483" path="/var/lib/kubelet/pods/7c4214b7-bfcf-4c83-ab47-c727e1a8a483/volumes" Jul 31 14:54:31.071325 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:31.071286 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:54:35.416069 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:35.416030 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd"] Jul 31 14:54:35.416878 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:35.416237 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" podUID="fb56dcb2-ea04-4a31-a5c6-79a38d24a834" containerName="istio-proxy" containerID="cri-o://5e013bb783c31623fcc85b4564ea2869a0ad134c67691bf976d9f655be3e7d25" gracePeriod=30 Jul 31 14:54:35.537641 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:35.537606 2577 patch_prober.go:28] interesting pod/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 14:54:35.537815 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:35.537678 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" podUID="fb56dcb2-ea04-4a31-a5c6-79a38d24a834" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 14:54:40.653332 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.653308 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:54:40.666633 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.666604 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-credential-socket\") pod \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " Jul 31 14:54:40.666723 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.666643 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-workload-socket\") pod \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " Jul 31 14:54:40.666723 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.666687 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svxng\" (UniqueName: \"kubernetes.io/projected/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-kube-api-access-svxng\") pod \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " Jul 31 14:54:40.666792 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.666751 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-envoy\") pod \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " Jul 31 14:54:40.666792 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.666777 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-podinfo\") pod \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " Jul 31 14:54:40.666882 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.666803 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istiod-ca-cert\") pod \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " Jul 31 14:54:40.666882 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.666829 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-workload-certs\") pod \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " Jul 31 14:54:40.666994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.666892 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-data\") pod \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " Jul 31 14:54:40.666994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.666920 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-token\") pod \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\" (UID: \"fb56dcb2-ea04-4a31-a5c6-79a38d24a834\") " Jul 31 14:54:40.666994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.666949 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fb56dcb2-ea04-4a31-a5c6-79a38d24a834" (UID: "fb56dcb2-ea04-4a31-a5c6-79a38d24a834"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:54:40.667144 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.667127 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fb56dcb2-ea04-4a31-a5c6-79a38d24a834" (UID: "fb56dcb2-ea04-4a31-a5c6-79a38d24a834"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:54:40.667207 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.667161 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-credential-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:54:40.667262 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.667239 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-data" (OuterVolumeSpecName: "istio-data") pod "fb56dcb2-ea04-4a31-a5c6-79a38d24a834" (UID: "fb56dcb2-ea04-4a31-a5c6-79a38d24a834"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:54:40.667438 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.667400 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fb56dcb2-ea04-4a31-a5c6-79a38d24a834" (UID: "fb56dcb2-ea04-4a31-a5c6-79a38d24a834"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:54:40.667622 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.667594 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fb56dcb2-ea04-4a31-a5c6-79a38d24a834" (UID: "fb56dcb2-ea04-4a31-a5c6-79a38d24a834"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:54:40.668982 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.668956 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fb56dcb2-ea04-4a31-a5c6-79a38d24a834" (UID: "fb56dcb2-ea04-4a31-a5c6-79a38d24a834"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:54:40.669078 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.668967 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-kube-api-access-svxng" (OuterVolumeSpecName: "kube-api-access-svxng") pod "fb56dcb2-ea04-4a31-a5c6-79a38d24a834" (UID: "fb56dcb2-ea04-4a31-a5c6-79a38d24a834"). InnerVolumeSpecName "kube-api-access-svxng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:54:40.669182 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.669162 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-token" (OuterVolumeSpecName: "istio-token") pod "fb56dcb2-ea04-4a31-a5c6-79a38d24a834" (UID: "fb56dcb2-ea04-4a31-a5c6-79a38d24a834"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:54:40.669268 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.669246 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fb56dcb2-ea04-4a31-a5c6-79a38d24a834" (UID: "fb56dcb2-ea04-4a31-a5c6-79a38d24a834"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 14:54:40.768445 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.768359 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-envoy\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:54:40.768445 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.768387 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-podinfo\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:54:40.768445 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.768395 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istiod-ca-cert\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:54:40.768445 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.768403 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-workload-certs\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:54:40.768445 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.768411 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-data\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:54:40.768445 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.768419 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-istio-token\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:54:40.768445 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.768426 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-workload-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:54:40.768445 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:40.768434 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svxng\" (UniqueName: \"kubernetes.io/projected/fb56dcb2-ea04-4a31-a5c6-79a38d24a834-kube-api-access-svxng\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:54:41.490551 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:41.490516 2577 generic.go:356] "Generic (PLEG): container finished" podID="fb56dcb2-ea04-4a31-a5c6-79a38d24a834" containerID="5e013bb783c31623fcc85b4564ea2869a0ad134c67691bf976d9f655be3e7d25" exitCode=0 Jul 31 14:54:41.490751 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:41.490562 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" Jul 31 14:54:41.490751 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:41.490582 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" event={"ID":"fb56dcb2-ea04-4a31-a5c6-79a38d24a834","Type":"ContainerDied","Data":"5e013bb783c31623fcc85b4564ea2869a0ad134c67691bf976d9f655be3e7d25"} Jul 31 14:54:41.490751 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:41.490623 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd" event={"ID":"fb56dcb2-ea04-4a31-a5c6-79a38d24a834","Type":"ContainerDied","Data":"8e15f6243e639f64553ba80b6f647c490550f2b753949eef149b50efa1e6a0c6"} Jul 31 14:54:41.490751 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:41.490644 2577 scope.go:117] "RemoveContainer" containerID="5e013bb783c31623fcc85b4564ea2869a0ad134c67691bf976d9f655be3e7d25" Jul 31 14:54:41.499218 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:41.499200 2577 scope.go:117] "RemoveContainer" containerID="cc60301077af993d040cdebad498911f9ec906b141e6f630eca4a96fcfd189ac" Jul 31 14:54:41.506389 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:41.506374 2577 scope.go:117] "RemoveContainer" containerID="5e013bb783c31623fcc85b4564ea2869a0ad134c67691bf976d9f655be3e7d25" Jul 31 14:54:41.506645 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:54:41.506627 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e013bb783c31623fcc85b4564ea2869a0ad134c67691bf976d9f655be3e7d25\": container with ID starting with 5e013bb783c31623fcc85b4564ea2869a0ad134c67691bf976d9f655be3e7d25 not found: ID does not exist" containerID="5e013bb783c31623fcc85b4564ea2869a0ad134c67691bf976d9f655be3e7d25" Jul 31 14:54:41.506695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:41.506653 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e013bb783c31623fcc85b4564ea2869a0ad134c67691bf976d9f655be3e7d25"} err="failed to get container status \"5e013bb783c31623fcc85b4564ea2869a0ad134c67691bf976d9f655be3e7d25\": rpc error: code = NotFound desc = could not find container \"5e013bb783c31623fcc85b4564ea2869a0ad134c67691bf976d9f655be3e7d25\": container with ID starting with 5e013bb783c31623fcc85b4564ea2869a0ad134c67691bf976d9f655be3e7d25 not found: ID does not exist" Jul 31 14:54:41.506695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:41.506670 2577 scope.go:117] "RemoveContainer" containerID="cc60301077af993d040cdebad498911f9ec906b141e6f630eca4a96fcfd189ac" Jul 31 14:54:41.506925 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:54:41.506907 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc60301077af993d040cdebad498911f9ec906b141e6f630eca4a96fcfd189ac\": container with ID starting with cc60301077af993d040cdebad498911f9ec906b141e6f630eca4a96fcfd189ac not found: ID does not exist" containerID="cc60301077af993d040cdebad498911f9ec906b141e6f630eca4a96fcfd189ac" Jul 31 14:54:41.506975 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:41.506931 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc60301077af993d040cdebad498911f9ec906b141e6f630eca4a96fcfd189ac"} err="failed to get container status \"cc60301077af993d040cdebad498911f9ec906b141e6f630eca4a96fcfd189ac\": rpc error: code = NotFound desc = could not find container \"cc60301077af993d040cdebad498911f9ec906b141e6f630eca4a96fcfd189ac\": container with ID starting with cc60301077af993d040cdebad498911f9ec906b141e6f630eca4a96fcfd189ac not found: ID does not exist" Jul 31 14:54:41.509812 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:41.509792 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd"] Jul 31 14:54:41.513775 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:41.513756 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7f0814aa-openshift-default-5b46c7cd64-d5vtd"] Jul 31 14:54:42.429512 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:54:42.429479 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb56dcb2-ea04-4a31-a5c6-79a38d24a834" path="/var/lib/kubelet/pods/fb56dcb2-ea04-4a31-a5c6-79a38d24a834/volumes" Jul 31 14:55:07.274033 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:55:07.273996 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:56:28.573325 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:28.573295 2577 scope.go:117] "RemoveContainer" containerID="bcc84c26d8b95cd8fda8f51d3853ccb883f03dfa353bedb87cee895bd7bca1da" Jul 31 14:56:50.549564 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.549527 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk"] Jul 31 14:56:50.550047 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.549815 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb56dcb2-ea04-4a31-a5c6-79a38d24a834" containerName="istio-proxy" Jul 31 14:56:50.550047 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.549825 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb56dcb2-ea04-4a31-a5c6-79a38d24a834" containerName="istio-proxy" Jul 31 14:56:50.550047 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.549837 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad959dc-ed8c-4764-ac5b-9ea331a5a906" containerName="payload-pre-processing" Jul 31 14:56:50.550047 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.549861 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad959dc-ed8c-4764-ac5b-9ea331a5a906" containerName="payload-pre-processing" Jul 31 14:56:50.550047 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.549878 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c4214b7-bfcf-4c83-ab47-c727e1a8a483" containerName="payload-processing" Jul 31 14:56:50.550047 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.549884 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c4214b7-bfcf-4c83-ab47-c727e1a8a483" containerName="payload-processing" Jul 31 14:56:50.550047 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.549892 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb56dcb2-ea04-4a31-a5c6-79a38d24a834" containerName="istio-proxy" Jul 31 14:56:50.550047 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.549897 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb56dcb2-ea04-4a31-a5c6-79a38d24a834" containerName="istio-proxy" Jul 31 14:56:50.550047 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.549944 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb56dcb2-ea04-4a31-a5c6-79a38d24a834" containerName="istio-proxy" Jul 31 14:56:50.550047 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.549950 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad959dc-ed8c-4764-ac5b-9ea331a5a906" containerName="payload-pre-processing" Jul 31 14:56:50.550047 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.549957 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb56dcb2-ea04-4a31-a5c6-79a38d24a834" containerName="istio-proxy" Jul 31 14:56:50.550047 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.549962 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c4214b7-bfcf-4c83-ab47-c727e1a8a483" containerName="payload-processing" Jul 31 14:56:50.552784 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.552767 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.554647 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.554621 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-86b90147-openshift-default-dockercfg-hqlzr\"" Jul 31 14:56:50.563523 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.563498 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk"] Jul 31 14:56:50.640184 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.640148 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr5md\" (UniqueName: \"kubernetes.io/projected/8f463fb3-9d29-4cb7-a294-d95aba8bd526-kube-api-access-qr5md\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.640370 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.640194 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-podinfo\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.640370 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.640304 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-credential-socket\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.640471 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.640392 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istiod-ca-cert\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.640471 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.640426 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-envoy\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.640471 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.640448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-data\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.640579 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.640476 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-workload-socket\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.640579 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.640528 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-workload-certs\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.640647 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.640582 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-token\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.741219 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.741180 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istiod-ca-cert\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.741395 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.741232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-envoy\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.741395 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.741254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-data\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.741395 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.741274 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-workload-socket\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.741395 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.741294 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-workload-certs\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.741395 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.741326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-token\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.741395 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.741351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr5md\" (UniqueName: \"kubernetes.io/projected/8f463fb3-9d29-4cb7-a294-d95aba8bd526-kube-api-access-qr5md\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.741395 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.741384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-podinfo\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.741741 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.741419 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-credential-socket\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.741741 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.741652 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-data\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.741871 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.741781 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-workload-socket\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.741871 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.741806 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-workload-certs\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.741988 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.741968 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-credential-socket\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.742041 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.742012 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istiod-ca-cert\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.743565 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.743542 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-envoy\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.744157 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.744139 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-podinfo\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.749177 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.749151 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-token\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.749567 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.749545 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr5md\" (UniqueName: \"kubernetes.io/projected/8f463fb3-9d29-4cb7-a294-d95aba8bd526-kube-api-access-qr5md\") pod \"e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.862642 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.862570 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:50.992760 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.992724 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk"] Jul 31 14:56:50.996032 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:56:50.996004 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f463fb3_9d29_4cb7_a294_d95aba8bd526.slice/crio-f3546976a30d21acdb957bc80cdf390e8bfd521b8b7bea752a8b5eae58d68379 WatchSource:0}: Error finding container f3546976a30d21acdb957bc80cdf390e8bfd521b8b7bea752a8b5eae58d68379: Status 404 returned error can't find the container with id f3546976a30d21acdb957bc80cdf390e8bfd521b8b7bea752a8b5eae58d68379 Jul 31 14:56:50.998226 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.998188 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:56:50.998301 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.998262 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:56:50.998301 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:50.998292 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:56:51.024606 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:51.024583 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" event={"ID":"8f463fb3-9d29-4cb7-a294-d95aba8bd526","Type":"ContainerStarted","Data":"f3546976a30d21acdb957bc80cdf390e8bfd521b8b7bea752a8b5eae58d68379"} Jul 31 14:56:52.030299 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:52.030262 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" event={"ID":"8f463fb3-9d29-4cb7-a294-d95aba8bd526","Type":"ContainerStarted","Data":"39e5c0e57a893014be8c9de307015d9595c1c39496f547452fe318d5da0f794b"} Jul 31 14:56:52.048813 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:52.048762 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" podStartSLOduration=2.048745205 podStartE2EDuration="2.048745205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:56:52.046689845 +0000 UTC m=+2664.083512911" watchObservedRunningTime="2026-07-31 14:56:52.048745205 +0000 UTC m=+2664.085568272" Jul 31 14:56:52.072833 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:52.072801 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:56:52.863692 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:52.863651 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:52.868295 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:52.868266 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:53.033917 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:53.033888 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:53.034914 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:53.034895 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:56:55.704105 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.704073 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx"] Jul 31 14:56:55.707295 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.707279 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx" Jul 31 14:56:55.710241 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.710214 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-86b90147-dockercfg-sfdh9\"" Jul 31 14:56:55.710426 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.710406 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-86b90147\"" Jul 31 14:56:55.718327 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.718306 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx"] Jul 31 14:56:55.734773 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.734751 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm"] Jul 31 14:56:55.737696 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.737682 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm" Jul 31 14:56:55.748133 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.748110 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm"] Jul 31 14:56:55.785251 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.785228 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5ngz\" (UniqueName: \"kubernetes.io/projected/62a211f1-2a1a-4b63-9cea-8bf1e6c7a407-kube-api-access-w5ngz\") pod \"payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx\" (UID: \"62a211f1-2a1a-4b63-9cea-8bf1e6c7a407\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx" Jul 31 14:56:55.785379 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.785266 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a211f1-2a1a-4b63-9cea-8bf1e6c7a407-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx\" (UID: \"62a211f1-2a1a-4b63-9cea-8bf1e6c7a407\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx" Jul 31 14:56:55.886744 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.886713 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d78bad00-4fee-4a81-b0db-c9fc9a7fe744-plugins-config-volume\") pod \"payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm\" (UID: \"d78bad00-4fee-4a81-b0db-c9fc9a7fe744\") " pod="openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm" Jul 31 14:56:55.887010 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.886752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brqsf\" (UniqueName: \"kubernetes.io/projected/d78bad00-4fee-4a81-b0db-c9fc9a7fe744-kube-api-access-brqsf\") pod \"payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm\" (UID: \"d78bad00-4fee-4a81-b0db-c9fc9a7fe744\") " pod="openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm" Jul 31 14:56:55.887010 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.886823 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5ngz\" (UniqueName: \"kubernetes.io/projected/62a211f1-2a1a-4b63-9cea-8bf1e6c7a407-kube-api-access-w5ngz\") pod \"payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx\" (UID: \"62a211f1-2a1a-4b63-9cea-8bf1e6c7a407\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx" Jul 31 14:56:55.887010 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.886881 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a211f1-2a1a-4b63-9cea-8bf1e6c7a407-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx\" (UID: \"62a211f1-2a1a-4b63-9cea-8bf1e6c7a407\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx" Jul 31 14:56:55.887496 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.887477 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a211f1-2a1a-4b63-9cea-8bf1e6c7a407-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx\" (UID: \"62a211f1-2a1a-4b63-9cea-8bf1e6c7a407\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx" Jul 31 14:56:55.899050 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.899022 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5ngz\" (UniqueName: \"kubernetes.io/projected/62a211f1-2a1a-4b63-9cea-8bf1e6c7a407-kube-api-access-w5ngz\") pod \"payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx\" (UID: \"62a211f1-2a1a-4b63-9cea-8bf1e6c7a407\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx" Jul 31 14:56:55.987813 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.987731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d78bad00-4fee-4a81-b0db-c9fc9a7fe744-plugins-config-volume\") pod \"payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm\" (UID: \"d78bad00-4fee-4a81-b0db-c9fc9a7fe744\") " pod="openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm" Jul 31 14:56:55.987813 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.987783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brqsf\" (UniqueName: \"kubernetes.io/projected/d78bad00-4fee-4a81-b0db-c9fc9a7fe744-kube-api-access-brqsf\") pod \"payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm\" (UID: \"d78bad00-4fee-4a81-b0db-c9fc9a7fe744\") " pod="openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm" Jul 31 14:56:55.988455 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:55.988437 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d78bad00-4fee-4a81-b0db-c9fc9a7fe744-plugins-config-volume\") pod \"payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm\" (UID: \"d78bad00-4fee-4a81-b0db-c9fc9a7fe744\") " pod="openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm" Jul 31 14:56:56.001378 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:56.001351 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brqsf\" (UniqueName: \"kubernetes.io/projected/d78bad00-4fee-4a81-b0db-c9fc9a7fe744-kube-api-access-brqsf\") pod \"payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm\" (UID: \"d78bad00-4fee-4a81-b0db-c9fc9a7fe744\") " pod="openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm" Jul 31 14:56:56.018179 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:56.018155 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx" Jul 31 14:56:56.047880 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:56.047854 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm" Jul 31 14:56:56.156120 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:56.156091 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx"] Jul 31 14:56:56.159918 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:56:56.159876 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a211f1_2a1a_4b63_9cea_8bf1e6c7a407.slice/crio-bfc0a5c41089f4cdcf82c352154abcb47d1162443ee13e102a724240ec7f2847 WatchSource:0}: Error finding container bfc0a5c41089f4cdcf82c352154abcb47d1162443ee13e102a724240ec7f2847: Status 404 returned error can't find the container with id bfc0a5c41089f4cdcf82c352154abcb47d1162443ee13e102a724240ec7f2847 Jul 31 14:56:56.184878 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:56.184856 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm"] Jul 31 14:56:56.194993 ip-10-0-132-38 kubenswrapper[2577]: W0731 14:56:56.194963 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78bad00_4fee_4a81_b0db_c9fc9a7fe744.slice/crio-f1f470d612872723f4cad57fb0dd05e40ae3ff72a0faea4f3349ef5ff587208c WatchSource:0}: Error finding container f1f470d612872723f4cad57fb0dd05e40ae3ff72a0faea4f3349ef5ff587208c: Status 404 returned error can't find the container with id f1f470d612872723f4cad57fb0dd05e40ae3ff72a0faea4f3349ef5ff587208c Jul 31 14:56:57.051480 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:57.051444 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm" event={"ID":"d78bad00-4fee-4a81-b0db-c9fc9a7fe744","Type":"ContainerStarted","Data":"29ff4628ac77eab75f6319630124249aa08ddebad5d7c2aec18535fc591ac9c8"} Jul 31 14:56:57.051953 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:57.051498 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm" event={"ID":"d78bad00-4fee-4a81-b0db-c9fc9a7fe744","Type":"ContainerStarted","Data":"f1f470d612872723f4cad57fb0dd05e40ae3ff72a0faea4f3349ef5ff587208c"} Jul 31 14:56:57.051953 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:57.051523 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm" Jul 31 14:56:57.052872 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:57.052830 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx" event={"ID":"62a211f1-2a1a-4b63-9cea-8bf1e6c7a407","Type":"ContainerStarted","Data":"e27cc19c4ab7f82140d1b27b89ed460464c2c87570e8b4baf5afc9e9f351a174"} Jul 31 14:56:57.052994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:57.052889 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx" event={"ID":"62a211f1-2a1a-4b63-9cea-8bf1e6c7a407","Type":"ContainerStarted","Data":"bfc0a5c41089f4cdcf82c352154abcb47d1162443ee13e102a724240ec7f2847"} Jul 31 14:56:57.052994 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:57.052962 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx" Jul 31 14:56:57.066893 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:57.066831 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm" podStartSLOduration=2.066818141 podStartE2EDuration="2.066818141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:56:57.065117762 +0000 UTC m=+2669.101940830" watchObservedRunningTime="2026-07-31 14:56:57.066818141 +0000 UTC m=+2669.103641208" Jul 31 14:56:57.078892 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:56:57.078820 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx" podStartSLOduration=2.078802676 podStartE2EDuration="2.078802676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:56:57.078527882 +0000 UTC m=+2669.115350960" watchObservedRunningTime="2026-07-31 14:56:57.078802676 +0000 UTC m=+2669.115625743" Jul 31 14:57:00.280907 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:00.280869 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:57:07.098111 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:07.098078 2577 generic.go:356] "Generic (PLEG): container finished" podID="8f463fb3-9d29-4cb7-a294-d95aba8bd526" containerID="39e5c0e57a893014be8c9de307015d9595c1c39496f547452fe318d5da0f794b" exitCode=0 Jul 31 14:57:07.098537 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:07.098150 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" event={"ID":"8f463fb3-9d29-4cb7-a294-d95aba8bd526","Type":"ContainerDied","Data":"39e5c0e57a893014be8c9de307015d9595c1c39496f547452fe318d5da0f794b"} Jul 31 14:57:07.098537 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:07.098497 2577 scope.go:117] "RemoveContainer" containerID="39e5c0e57a893014be8c9de307015d9595c1c39496f547452fe318d5da0f794b" Jul 31 14:57:07.099402 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:07.099376 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:57:07.099462 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:07.099440 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:57:07.099542 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:07.099470 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 14:57:08.044863 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:08.043957 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:57:08.060308 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:08.060276 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm" Jul 31 14:57:08.060513 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:08.060355 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx" Jul 31 14:57:08.106428 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:08.106397 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" event={"ID":"8f463fb3-9d29-4cb7-a294-d95aba8bd526","Type":"ContainerStarted","Data":"0e11dd2a0c53a12883cd0c11985a2c3cd25850e7170b80751d174623bba5403f"} Jul 31 14:57:08.863437 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:08.863408 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:57:08.864651 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:08.864627 2577 patch_prober.go:28] interesting pod/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 31 14:57:08.864760 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:08.864682 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" podUID="8f463fb3-9d29-4cb7-a294-d95aba8bd526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 31 14:57:09.863459 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:09.863421 2577 patch_prober.go:28] interesting pod/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 31 14:57:09.863870 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:09.863480 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" podUID="8f463fb3-9d29-4cb7-a294-d95aba8bd526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 31 14:57:10.863947 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:10.863913 2577 patch_prober.go:28] interesting pod/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 31 14:57:10.864350 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:10.863966 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" podUID="8f463fb3-9d29-4cb7-a294-d95aba8bd526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 31 14:57:11.863560 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:11.863529 2577 patch_prober.go:28] interesting pod/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 31 14:57:11.863739 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:11.863583 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" podUID="8f463fb3-9d29-4cb7-a294-d95aba8bd526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 31 14:57:12.863582 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:12.863544 2577 patch_prober.go:28] interesting pod/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 31 14:57:12.864021 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:12.863608 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" podUID="8f463fb3-9d29-4cb7-a294-d95aba8bd526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 31 14:57:13.867693 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:13.867665 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:57:13.868071 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:13.867937 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:57:13.868661 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:13.868642 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:57:28.589797 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:28.589723 2577 scope.go:117] "RemoveContainer" containerID="8444e86bf4e26713f2202444af8132293712a92975125c7bf439ab0368fbffde" Jul 31 14:57:32.372070 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:32.372039 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:57:47.984942 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:57:47.984907 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:58:11.975429 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:11.975392 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:58:14.487022 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:14.486983 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm"] Jul 31 14:58:14.487497 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:14.487184 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm" podUID="d78bad00-4fee-4a81-b0db-c9fc9a7fe744" containerName="payload-processing" containerID="cri-o://29ff4628ac77eab75f6319630124249aa08ddebad5d7c2aec18535fc591ac9c8" gracePeriod=30 Jul 31 14:58:14.493873 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:14.493828 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx"] Jul 31 14:58:14.494406 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:14.494072 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx" podUID="62a211f1-2a1a-4b63-9cea-8bf1e6c7a407" containerName="payload-pre-processing" containerID="cri-o://e27cc19c4ab7f82140d1b27b89ed460464c2c87570e8b4baf5afc9e9f351a174" gracePeriod=30 Jul 31 14:58:14.765637 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:14.765617 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm" Jul 31 14:58:14.768162 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:14.768146 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx" Jul 31 14:58:14.853377 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:14.853351 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5ngz\" (UniqueName: \"kubernetes.io/projected/62a211f1-2a1a-4b63-9cea-8bf1e6c7a407-kube-api-access-w5ngz\") pod \"62a211f1-2a1a-4b63-9cea-8bf1e6c7a407\" (UID: \"62a211f1-2a1a-4b63-9cea-8bf1e6c7a407\") " Jul 31 14:58:14.853533 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:14.853403 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d78bad00-4fee-4a81-b0db-c9fc9a7fe744-plugins-config-volume\") pod \"d78bad00-4fee-4a81-b0db-c9fc9a7fe744\" (UID: \"d78bad00-4fee-4a81-b0db-c9fc9a7fe744\") " Jul 31 14:58:14.853533 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:14.853435 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brqsf\" (UniqueName: \"kubernetes.io/projected/d78bad00-4fee-4a81-b0db-c9fc9a7fe744-kube-api-access-brqsf\") pod \"d78bad00-4fee-4a81-b0db-c9fc9a7fe744\" (UID: \"d78bad00-4fee-4a81-b0db-c9fc9a7fe744\") " Jul 31 14:58:14.853533 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:14.853486 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a211f1-2a1a-4b63-9cea-8bf1e6c7a407-plugins-config-volume\") pod \"62a211f1-2a1a-4b63-9cea-8bf1e6c7a407\" (UID: \"62a211f1-2a1a-4b63-9cea-8bf1e6c7a407\") " Jul 31 14:58:14.853902 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:14.853875 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d78bad00-4fee-4a81-b0db-c9fc9a7fe744-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d78bad00-4fee-4a81-b0db-c9fc9a7fe744" (UID: "d78bad00-4fee-4a81-b0db-c9fc9a7fe744"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:58:14.853996 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:14.853902 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a211f1-2a1a-4b63-9cea-8bf1e6c7a407-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62a211f1-2a1a-4b63-9cea-8bf1e6c7a407" (UID: "62a211f1-2a1a-4b63-9cea-8bf1e6c7a407"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:58:14.855485 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:14.855466 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a211f1-2a1a-4b63-9cea-8bf1e6c7a407-kube-api-access-w5ngz" (OuterVolumeSpecName: "kube-api-access-w5ngz") pod "62a211f1-2a1a-4b63-9cea-8bf1e6c7a407" (UID: "62a211f1-2a1a-4b63-9cea-8bf1e6c7a407"). InnerVolumeSpecName "kube-api-access-w5ngz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:58:14.855581 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:14.855498 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78bad00-4fee-4a81-b0db-c9fc9a7fe744-kube-api-access-brqsf" (OuterVolumeSpecName: "kube-api-access-brqsf") pod "d78bad00-4fee-4a81-b0db-c9fc9a7fe744" (UID: "d78bad00-4fee-4a81-b0db-c9fc9a7fe744"). InnerVolumeSpecName "kube-api-access-brqsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:58:14.955064 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:14.955032 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a211f1-2a1a-4b63-9cea-8bf1e6c7a407-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:58:14.955064 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:14.955059 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5ngz\" (UniqueName: \"kubernetes.io/projected/62a211f1-2a1a-4b63-9cea-8bf1e6c7a407-kube-api-access-w5ngz\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:58:14.955064 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:14.955068 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d78bad00-4fee-4a81-b0db-c9fc9a7fe744-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:58:14.955240 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:14.955077 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brqsf\" (UniqueName: \"kubernetes.io/projected/d78bad00-4fee-4a81-b0db-c9fc9a7fe744-kube-api-access-brqsf\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:58:15.394535 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:15.394495 2577 generic.go:356] "Generic (PLEG): container finished" podID="d78bad00-4fee-4a81-b0db-c9fc9a7fe744" containerID="29ff4628ac77eab75f6319630124249aa08ddebad5d7c2aec18535fc591ac9c8" exitCode=0 Jul 31 14:58:15.394709 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:15.394540 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm" Jul 31 14:58:15.394709 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:15.394580 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm" event={"ID":"d78bad00-4fee-4a81-b0db-c9fc9a7fe744","Type":"ContainerDied","Data":"29ff4628ac77eab75f6319630124249aa08ddebad5d7c2aec18535fc591ac9c8"} Jul 31 14:58:15.394709 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:15.394634 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm" event={"ID":"d78bad00-4fee-4a81-b0db-c9fc9a7fe744","Type":"ContainerDied","Data":"f1f470d612872723f4cad57fb0dd05e40ae3ff72a0faea4f3349ef5ff587208c"} Jul 31 14:58:15.394709 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:15.394663 2577 scope.go:117] "RemoveContainer" containerID="29ff4628ac77eab75f6319630124249aa08ddebad5d7c2aec18535fc591ac9c8" Jul 31 14:58:15.395914 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:15.395894 2577 generic.go:356] "Generic (PLEG): container finished" podID="62a211f1-2a1a-4b63-9cea-8bf1e6c7a407" containerID="e27cc19c4ab7f82140d1b27b89ed460464c2c87570e8b4baf5afc9e9f351a174" exitCode=0 Jul 31 14:58:15.396009 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:15.395934 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx" Jul 31 14:58:15.396009 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:15.395948 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx" event={"ID":"62a211f1-2a1a-4b63-9cea-8bf1e6c7a407","Type":"ContainerDied","Data":"e27cc19c4ab7f82140d1b27b89ed460464c2c87570e8b4baf5afc9e9f351a174"} Jul 31 14:58:15.396009 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:15.395980 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx" event={"ID":"62a211f1-2a1a-4b63-9cea-8bf1e6c7a407","Type":"ContainerDied","Data":"bfc0a5c41089f4cdcf82c352154abcb47d1162443ee13e102a724240ec7f2847"} Jul 31 14:58:15.404570 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:15.404552 2577 scope.go:117] "RemoveContainer" containerID="29ff4628ac77eab75f6319630124249aa08ddebad5d7c2aec18535fc591ac9c8" Jul 31 14:58:15.404868 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:58:15.404822 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ff4628ac77eab75f6319630124249aa08ddebad5d7c2aec18535fc591ac9c8\": container with ID starting with 29ff4628ac77eab75f6319630124249aa08ddebad5d7c2aec18535fc591ac9c8 not found: ID does not exist" containerID="29ff4628ac77eab75f6319630124249aa08ddebad5d7c2aec18535fc591ac9c8" Jul 31 14:58:15.404963 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:15.404869 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ff4628ac77eab75f6319630124249aa08ddebad5d7c2aec18535fc591ac9c8"} err="failed to get container status \"29ff4628ac77eab75f6319630124249aa08ddebad5d7c2aec18535fc591ac9c8\": rpc error: code = NotFound desc = could not find container \"29ff4628ac77eab75f6319630124249aa08ddebad5d7c2aec18535fc591ac9c8\": container with ID starting with 29ff4628ac77eab75f6319630124249aa08ddebad5d7c2aec18535fc591ac9c8 not found: ID does not exist" Jul 31 14:58:15.404963 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:15.404887 2577 scope.go:117] "RemoveContainer" containerID="e27cc19c4ab7f82140d1b27b89ed460464c2c87570e8b4baf5afc9e9f351a174" Jul 31 14:58:15.412228 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:15.412210 2577 scope.go:117] "RemoveContainer" containerID="e27cc19c4ab7f82140d1b27b89ed460464c2c87570e8b4baf5afc9e9f351a174" Jul 31 14:58:15.412453 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:58:15.412435 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e27cc19c4ab7f82140d1b27b89ed460464c2c87570e8b4baf5afc9e9f351a174\": container with ID starting with e27cc19c4ab7f82140d1b27b89ed460464c2c87570e8b4baf5afc9e9f351a174 not found: ID does not exist" containerID="e27cc19c4ab7f82140d1b27b89ed460464c2c87570e8b4baf5afc9e9f351a174" Jul 31 14:58:15.412504 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:15.412461 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e27cc19c4ab7f82140d1b27b89ed460464c2c87570e8b4baf5afc9e9f351a174"} err="failed to get container status \"e27cc19c4ab7f82140d1b27b89ed460464c2c87570e8b4baf5afc9e9f351a174\": rpc error: code = NotFound desc = could not find container \"e27cc19c4ab7f82140d1b27b89ed460464c2c87570e8b4baf5afc9e9f351a174\": container with ID starting with e27cc19c4ab7f82140d1b27b89ed460464c2c87570e8b4baf5afc9e9f351a174 not found: ID does not exist" Jul 31 14:58:15.414729 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:15.414709 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm"] Jul 31 14:58:15.421817 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:15.421798 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-86b90147-694b9fc78b-6lzcm"] Jul 31 14:58:15.431048 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:15.431031 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx"] Jul 31 14:58:15.435250 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:15.435233 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-86b90147-6c6c87d7cb-82dpx"] Jul 31 14:58:16.429158 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:16.429120 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62a211f1-2a1a-4b63-9cea-8bf1e6c7a407" path="/var/lib/kubelet/pods/62a211f1-2a1a-4b63-9cea-8bf1e6c7a407/volumes" Jul 31 14:58:16.429537 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:16.429473 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d78bad00-4fee-4a81-b0db-c9fc9a7fe744" path="/var/lib/kubelet/pods/d78bad00-4fee-4a81-b0db-c9fc9a7fe744/volumes" Jul 31 14:58:17.870481 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:58:17.870443 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:59:05.941105 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:05.941072 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk"] Jul 31 14:59:05.941667 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:05.941297 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" podUID="8f463fb3-9d29-4cb7-a294-d95aba8bd526" containerName="istio-proxy" containerID="cri-o://0e11dd2a0c53a12883cd0c11985a2c3cd25850e7170b80751d174623bba5403f" gracePeriod=30 Jul 31 14:59:08.071452 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:08.071416 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:59:11.185226 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.185202 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:59:11.223392 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.223316 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr5md\" (UniqueName: \"kubernetes.io/projected/8f463fb3-9d29-4cb7-a294-d95aba8bd526-kube-api-access-qr5md\") pod \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " Jul 31 14:59:11.223392 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.223382 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istiod-ca-cert\") pod \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " Jul 31 14:59:11.223597 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.223430 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-token\") pod \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " Jul 31 14:59:11.223597 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.223486 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-podinfo\") pod \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " Jul 31 14:59:11.223597 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.223516 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-data\") pod \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " Jul 31 14:59:11.223597 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.223543 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-workload-certs\") pod \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " Jul 31 14:59:11.223597 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.223580 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-credential-socket\") pod \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " Jul 31 14:59:11.223952 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.223743 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-envoy\") pod \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " Jul 31 14:59:11.223952 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.223781 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-workload-socket\") pod \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\" (UID: \"8f463fb3-9d29-4cb7-a294-d95aba8bd526\") " Jul 31 14:59:11.223952 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.223872 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-data" (OuterVolumeSpecName: "istio-data") pod "8f463fb3-9d29-4cb7-a294-d95aba8bd526" (UID: "8f463fb3-9d29-4cb7-a294-d95aba8bd526"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:59:11.223952 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.223913 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8f463fb3-9d29-4cb7-a294-d95aba8bd526" (UID: "8f463fb3-9d29-4cb7-a294-d95aba8bd526"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:59:11.224759 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.224599 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8f463fb3-9d29-4cb7-a294-d95aba8bd526" (UID: "8f463fb3-9d29-4cb7-a294-d95aba8bd526"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:59:11.224759 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.224707 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8f463fb3-9d29-4cb7-a294-d95aba8bd526" (UID: "8f463fb3-9d29-4cb7-a294-d95aba8bd526"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:59:11.225085 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.224989 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-credential-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:59:11.225085 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.225020 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-workload-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:59:11.225085 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.225037 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-data\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:59:11.225085 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.225062 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-workload-certs\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:59:11.231320 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.225519 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8f463fb3-9d29-4cb7-a294-d95aba8bd526" (UID: "8f463fb3-9d29-4cb7-a294-d95aba8bd526"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:59:11.231570 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.231503 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f463fb3-9d29-4cb7-a294-d95aba8bd526-kube-api-access-qr5md" (OuterVolumeSpecName: "kube-api-access-qr5md") pod "8f463fb3-9d29-4cb7-a294-d95aba8bd526" (UID: "8f463fb3-9d29-4cb7-a294-d95aba8bd526"). InnerVolumeSpecName "kube-api-access-qr5md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:59:11.231917 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.231892 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8f463fb3-9d29-4cb7-a294-d95aba8bd526" (UID: "8f463fb3-9d29-4cb7-a294-d95aba8bd526"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:59:11.232034 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.231913 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-token" (OuterVolumeSpecName: "istio-token") pod "8f463fb3-9d29-4cb7-a294-d95aba8bd526" (UID: "8f463fb3-9d29-4cb7-a294-d95aba8bd526"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:59:11.232103 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.232060 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8f463fb3-9d29-4cb7-a294-d95aba8bd526" (UID: "8f463fb3-9d29-4cb7-a294-d95aba8bd526"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 14:59:11.325967 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.325940 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-podinfo\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:59:11.325967 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.325965 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-envoy\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:59:11.326128 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.325975 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr5md\" (UniqueName: \"kubernetes.io/projected/8f463fb3-9d29-4cb7-a294-d95aba8bd526-kube-api-access-qr5md\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:59:11.326128 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.325985 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istiod-ca-cert\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:59:11.326128 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.325993 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f463fb3-9d29-4cb7-a294-d95aba8bd526-istio-token\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 14:59:11.626513 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.626425 2577 generic.go:356] "Generic (PLEG): container finished" podID="8f463fb3-9d29-4cb7-a294-d95aba8bd526" containerID="0e11dd2a0c53a12883cd0c11985a2c3cd25850e7170b80751d174623bba5403f" exitCode=0 Jul 31 14:59:11.626513 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.626487 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" Jul 31 14:59:11.626695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.626512 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" event={"ID":"8f463fb3-9d29-4cb7-a294-d95aba8bd526","Type":"ContainerDied","Data":"0e11dd2a0c53a12883cd0c11985a2c3cd25850e7170b80751d174623bba5403f"} Jul 31 14:59:11.626695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.626568 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk" event={"ID":"8f463fb3-9d29-4cb7-a294-d95aba8bd526","Type":"ContainerDied","Data":"f3546976a30d21acdb957bc80cdf390e8bfd521b8b7bea752a8b5eae58d68379"} Jul 31 14:59:11.626695 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.626591 2577 scope.go:117] "RemoveContainer" containerID="0e11dd2a0c53a12883cd0c11985a2c3cd25850e7170b80751d174623bba5403f" Jul 31 14:59:11.635543 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.635527 2577 scope.go:117] "RemoveContainer" containerID="39e5c0e57a893014be8c9de307015d9595c1c39496f547452fe318d5da0f794b" Jul 31 14:59:11.642811 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.642797 2577 scope.go:117] "RemoveContainer" containerID="0e11dd2a0c53a12883cd0c11985a2c3cd25850e7170b80751d174623bba5403f" Jul 31 14:59:11.643058 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:59:11.643042 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e11dd2a0c53a12883cd0c11985a2c3cd25850e7170b80751d174623bba5403f\": container with ID starting with 0e11dd2a0c53a12883cd0c11985a2c3cd25850e7170b80751d174623bba5403f not found: ID does not exist" containerID="0e11dd2a0c53a12883cd0c11985a2c3cd25850e7170b80751d174623bba5403f" Jul 31 14:59:11.643109 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.643065 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e11dd2a0c53a12883cd0c11985a2c3cd25850e7170b80751d174623bba5403f"} err="failed to get container status \"0e11dd2a0c53a12883cd0c11985a2c3cd25850e7170b80751d174623bba5403f\": rpc error: code = NotFound desc = could not find container \"0e11dd2a0c53a12883cd0c11985a2c3cd25850e7170b80751d174623bba5403f\": container with ID starting with 0e11dd2a0c53a12883cd0c11985a2c3cd25850e7170b80751d174623bba5403f not found: ID does not exist" Jul 31 14:59:11.643109 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.643080 2577 scope.go:117] "RemoveContainer" containerID="39e5c0e57a893014be8c9de307015d9595c1c39496f547452fe318d5da0f794b" Jul 31 14:59:11.643309 ip-10-0-132-38 kubenswrapper[2577]: E0731 14:59:11.643290 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39e5c0e57a893014be8c9de307015d9595c1c39496f547452fe318d5da0f794b\": container with ID starting with 39e5c0e57a893014be8c9de307015d9595c1c39496f547452fe318d5da0f794b not found: ID does not exist" containerID="39e5c0e57a893014be8c9de307015d9595c1c39496f547452fe318d5da0f794b" Jul 31 14:59:11.643368 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.643318 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39e5c0e57a893014be8c9de307015d9595c1c39496f547452fe318d5da0f794b"} err="failed to get container status \"39e5c0e57a893014be8c9de307015d9595c1c39496f547452fe318d5da0f794b\": rpc error: code = NotFound desc = could not find container \"39e5c0e57a893014be8c9de307015d9595c1c39496f547452fe318d5da0f794b\": container with ID starting with 39e5c0e57a893014be8c9de307015d9595c1c39496f547452fe318d5da0f794b not found: ID does not exist" Jul 31 14:59:11.647275 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.647250 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk"] Jul 31 14:59:11.650766 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:11.650747 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-86b90147-openshift-default-5846f58fb-lv7dk"] Jul 31 14:59:12.429210 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:12.429168 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f463fb3-9d29-4cb7-a294-d95aba8bd526" path="/var/lib/kubelet/pods/8f463fb3-9d29-4cb7-a294-d95aba8bd526/volumes" Jul 31 14:59:20.279866 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:20.279826 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 14:59:27.177827 ip-10-0-132-38 kubenswrapper[2577]: I0731 14:59:27.177791 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:00:00.137972 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.137933 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758500-mbqrf"] Jul 31 15:00:00.138480 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.138218 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a211f1-2a1a-4b63-9cea-8bf1e6c7a407" containerName="payload-pre-processing" Jul 31 15:00:00.138480 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.138228 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a211f1-2a1a-4b63-9cea-8bf1e6c7a407" containerName="payload-pre-processing" Jul 31 15:00:00.138480 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.138246 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f463fb3-9d29-4cb7-a294-d95aba8bd526" containerName="istio-proxy" Jul 31 15:00:00.138480 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.138252 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f463fb3-9d29-4cb7-a294-d95aba8bd526" containerName="istio-proxy" Jul 31 15:00:00.138480 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.138259 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f463fb3-9d29-4cb7-a294-d95aba8bd526" containerName="istio-proxy" Jul 31 15:00:00.138480 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.138264 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f463fb3-9d29-4cb7-a294-d95aba8bd526" containerName="istio-proxy" Jul 31 15:00:00.138480 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.138271 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d78bad00-4fee-4a81-b0db-c9fc9a7fe744" containerName="payload-processing" Jul 31 15:00:00.138480 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.138277 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d78bad00-4fee-4a81-b0db-c9fc9a7fe744" containerName="payload-processing" Jul 31 15:00:00.138480 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.138341 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f463fb3-9d29-4cb7-a294-d95aba8bd526" containerName="istio-proxy" Jul 31 15:00:00.138480 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.138349 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f463fb3-9d29-4cb7-a294-d95aba8bd526" containerName="istio-proxy" Jul 31 15:00:00.138480 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.138356 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a211f1-2a1a-4b63-9cea-8bf1e6c7a407" containerName="payload-pre-processing" Jul 31 15:00:00.138480 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.138362 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d78bad00-4fee-4a81-b0db-c9fc9a7fe744" containerName="payload-processing" Jul 31 15:00:00.141281 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.141264 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758500-mbqrf" Jul 31 15:00:00.144188 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.144163 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\"" Jul 31 15:00:00.150967 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.150944 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758500-mbqrf" Jul 31 15:00:00.152214 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.152195 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758500-mbqrf"] Jul 31 15:00:00.277904 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.277825 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758500-mbqrf"] Jul 31 15:00:00.280476 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:00:00.280443 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde570bb7_92e1_484c_9ef9_cd1ca5b4d6d8.slice/crio-03c60b74a1ffbb203bfbb264cbeefdb812e929708a68c0c724e187f058be884c WatchSource:0}: Error finding container 03c60b74a1ffbb203bfbb264cbeefdb812e929708a68c0c724e187f058be884c: Status 404 returned error can't find the container with id 03c60b74a1ffbb203bfbb264cbeefdb812e929708a68c0c724e187f058be884c Jul 31 15:00:00.828815 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.828718 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758500-mbqrf" event={"ID":"de570bb7-92e1-484c-9ef9-cd1ca5b4d6d8","Type":"ContainerStarted","Data":"3592d81ccd7b4db9851589323a811d4e109cf7ca4cb03a8ffad8b6cdbe540191"} Jul 31 15:00:00.828815 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.828781 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758500-mbqrf" event={"ID":"de570bb7-92e1-484c-9ef9-cd1ca5b4d6d8","Type":"ContainerStarted","Data":"03c60b74a1ffbb203bfbb264cbeefdb812e929708a68c0c724e187f058be884c"} Jul 31 15:00:00.842514 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:00.842465 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758500-mbqrf" podStartSLOduration=0.842451558 podStartE2EDuration="842.451558ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:00:00.841432097 +0000 UTC m=+2852.878255165" watchObservedRunningTime="2026-07-31 15:00:00.842451558 +0000 UTC m=+2852.879274628" Jul 31 15:00:01.834588 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:01.834551 2577 generic.go:356] "Generic (PLEG): container finished" podID="de570bb7-92e1-484c-9ef9-cd1ca5b4d6d8" containerID="3592d81ccd7b4db9851589323a811d4e109cf7ca4cb03a8ffad8b6cdbe540191" exitCode=0 Jul 31 15:00:01.834588 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:01.834593 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758500-mbqrf" event={"ID":"de570bb7-92e1-484c-9ef9-cd1ca5b4d6d8","Type":"ContainerDied","Data":"3592d81ccd7b4db9851589323a811d4e109cf7ca4cb03a8ffad8b6cdbe540191"} Jul 31 15:00:02.963830 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:02.963801 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758500-mbqrf" Jul 31 15:00:03.844165 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:03.844123 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758500-mbqrf" event={"ID":"de570bb7-92e1-484c-9ef9-cd1ca5b4d6d8","Type":"ContainerDied","Data":"03c60b74a1ffbb203bfbb264cbeefdb812e929708a68c0c724e187f058be884c"} Jul 31 15:00:03.844367 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:03.844178 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03c60b74a1ffbb203bfbb264cbeefdb812e929708a68c0c724e187f058be884c" Jul 31 15:00:03.844367 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:03.844137 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758500-mbqrf" Jul 31 15:00:11.208553 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.208516 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m"] Jul 31 15:00:11.209042 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.208967 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de570bb7-92e1-484c-9ef9-cd1ca5b4d6d8" containerName="cleanup" Jul 31 15:00:11.209042 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.208986 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="de570bb7-92e1-484c-9ef9-cd1ca5b4d6d8" containerName="cleanup" Jul 31 15:00:11.209176 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.209069 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="de570bb7-92e1-484c-9ef9-cd1ca5b4d6d8" containerName="cleanup" Jul 31 15:00:11.211897 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.211875 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.213708 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.213691 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-55959d-openshift-default-dockercfg-8kqvh\"" Jul 31 15:00:11.223725 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.223707 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m"] Jul 31 15:00:11.315944 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.315916 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89be0d85-23f1-4da0-ad33-feabc2518bfa-istiod-ca-cert\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.316112 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.315954 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-workload-certs\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.316112 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.315992 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-data\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.316112 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.316016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-workload-socket\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.316112 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.316040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-podinfo\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.316112 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.316055 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8gfk\" (UniqueName: \"kubernetes.io/projected/89be0d85-23f1-4da0-ad33-feabc2518bfa-kube-api-access-k8gfk\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.316112 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.316076 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-envoy\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.316422 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.316134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-token\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.316422 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.316159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-credential-socket\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.416795 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.416760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-podinfo\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.416795 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.416794 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8gfk\" (UniqueName: \"kubernetes.io/projected/89be0d85-23f1-4da0-ad33-feabc2518bfa-kube-api-access-k8gfk\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.417049 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.416815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-envoy\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.417049 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.416835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-token\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.417049 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.416879 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-credential-socket\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.417049 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.416907 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89be0d85-23f1-4da0-ad33-feabc2518bfa-istiod-ca-cert\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.417049 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.416930 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-workload-certs\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.417049 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.416964 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-data\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.417049 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.416986 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-workload-socket\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.417403 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.417364 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-workload-certs\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.417403 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.417380 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-workload-socket\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.417508 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.417441 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-data\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.417562 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.417516 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-credential-socket\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.417714 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.417695 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89be0d85-23f1-4da0-ad33-feabc2518bfa-istiod-ca-cert\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.419609 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.419586 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-envoy\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.419941 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.419921 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-podinfo\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.424565 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.424543 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-token\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.425007 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.424987 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8gfk\" (UniqueName: \"kubernetes.io/projected/89be0d85-23f1-4da0-ad33-feabc2518bfa-kube-api-access-k8gfk\") pod \"e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.521865 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.521778 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:11.655273 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.655238 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m"] Jul 31 15:00:11.659150 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:00:11.659123 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89be0d85_23f1_4da0_ad33_feabc2518bfa.slice/crio-64d29db13db81340b4cb18079f77997abaee012c371d14f8f3e529811474cdc1 WatchSource:0}: Error finding container 64d29db13db81340b4cb18079f77997abaee012c371d14f8f3e529811474cdc1: Status 404 returned error can't find the container with id 64d29db13db81340b4cb18079f77997abaee012c371d14f8f3e529811474cdc1 Jul 31 15:00:11.661279 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.661244 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:00:11.661399 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.661323 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:00:11.661399 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.661370 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:00:11.877294 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.877204 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" event={"ID":"89be0d85-23f1-4da0-ad33-feabc2518bfa","Type":"ContainerStarted","Data":"1b8b7d42f57c7d05d5e7edf1213da5950498874b37f1ce9ab246cfe4dec50cde"} Jul 31 15:00:11.877294 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.877267 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" event={"ID":"89be0d85-23f1-4da0-ad33-feabc2518bfa","Type":"ContainerStarted","Data":"64d29db13db81340b4cb18079f77997abaee012c371d14f8f3e529811474cdc1"} Jul 31 15:00:11.898800 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:11.898747 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" podStartSLOduration=0.898731174 podStartE2EDuration="898.731174ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:00:11.896629223 +0000 UTC m=+2863.933452291" watchObservedRunningTime="2026-07-31 15:00:11.898731174 +0000 UTC m=+2863.935554244" Jul 31 15:00:12.522918 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:12.522883 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:12.527666 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:12.527638 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:12.881784 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:12.881757 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:12.882870 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:12.882835 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:15.719832 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.719797 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn"] Jul 31 15:00:15.722604 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.722587 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn" Jul 31 15:00:15.726407 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.726382 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-55959d-dockercfg-thrbb\"" Jul 31 15:00:15.726773 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.726752 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-55959d\"" Jul 31 15:00:15.741430 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.741403 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn"] Jul 31 15:00:15.758197 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.758171 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m"] Jul 31 15:00:15.761231 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.761218 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m" Jul 31 15:00:15.768575 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.768547 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m"] Jul 31 15:00:15.856283 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.856248 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5l7d\" (UniqueName: \"kubernetes.io/projected/70a7fb16-5c0d-4088-b0d0-ec91d5185db0-kube-api-access-j5l7d\") pod \"payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn\" (UID: \"70a7fb16-5c0d-4088-b0d0-ec91d5185db0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn" Jul 31 15:00:15.856443 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.856292 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l77cr\" (UniqueName: \"kubernetes.io/projected/8377efc7-9f86-4d3f-a88f-2697fc804f53-kube-api-access-l77cr\") pod \"payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m\" (UID: \"8377efc7-9f86-4d3f-a88f-2697fc804f53\") " pod="openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m" Jul 31 15:00:15.856443 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.856415 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8377efc7-9f86-4d3f-a88f-2697fc804f53-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m\" (UID: \"8377efc7-9f86-4d3f-a88f-2697fc804f53\") " pod="openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m" Jul 31 15:00:15.856514 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.856445 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a7fb16-5c0d-4088-b0d0-ec91d5185db0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn\" (UID: \"70a7fb16-5c0d-4088-b0d0-ec91d5185db0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn" Jul 31 15:00:15.957157 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.957125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5l7d\" (UniqueName: \"kubernetes.io/projected/70a7fb16-5c0d-4088-b0d0-ec91d5185db0-kube-api-access-j5l7d\") pod \"payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn\" (UID: \"70a7fb16-5c0d-4088-b0d0-ec91d5185db0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn" Jul 31 15:00:15.957355 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.957173 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l77cr\" (UniqueName: \"kubernetes.io/projected/8377efc7-9f86-4d3f-a88f-2697fc804f53-kube-api-access-l77cr\") pod \"payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m\" (UID: \"8377efc7-9f86-4d3f-a88f-2697fc804f53\") " pod="openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m" Jul 31 15:00:15.957355 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.957230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8377efc7-9f86-4d3f-a88f-2697fc804f53-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m\" (UID: \"8377efc7-9f86-4d3f-a88f-2697fc804f53\") " pod="openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m" Jul 31 15:00:15.957355 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.957247 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a7fb16-5c0d-4088-b0d0-ec91d5185db0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn\" (UID: \"70a7fb16-5c0d-4088-b0d0-ec91d5185db0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn" Jul 31 15:00:15.957949 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.957930 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8377efc7-9f86-4d3f-a88f-2697fc804f53-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m\" (UID: \"8377efc7-9f86-4d3f-a88f-2697fc804f53\") " pod="openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m" Jul 31 15:00:15.958025 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.957983 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a7fb16-5c0d-4088-b0d0-ec91d5185db0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn\" (UID: \"70a7fb16-5c0d-4088-b0d0-ec91d5185db0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn" Jul 31 15:00:15.964580 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.964554 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l77cr\" (UniqueName: \"kubernetes.io/projected/8377efc7-9f86-4d3f-a88f-2697fc804f53-kube-api-access-l77cr\") pod \"payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m\" (UID: \"8377efc7-9f86-4d3f-a88f-2697fc804f53\") " pod="openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m" Jul 31 15:00:15.964691 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:15.964672 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5l7d\" (UniqueName: \"kubernetes.io/projected/70a7fb16-5c0d-4088-b0d0-ec91d5185db0-kube-api-access-j5l7d\") pod \"payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn\" (UID: \"70a7fb16-5c0d-4088-b0d0-ec91d5185db0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn" Jul 31 15:00:16.032752 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:16.032693 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn" Jul 31 15:00:16.071299 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:16.071259 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m" Jul 31 15:00:16.168354 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:16.167910 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn"] Jul 31 15:00:16.170607 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:00:16.170576 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a7fb16_5c0d_4088_b0d0_ec91d5185db0.slice/crio-6afa519c137d6a3e1b2c0dbdb9c94c7089a82e94866e35507fddda959f6ea2f7 WatchSource:0}: Error finding container 6afa519c137d6a3e1b2c0dbdb9c94c7089a82e94866e35507fddda959f6ea2f7: Status 404 returned error can't find the container with id 6afa519c137d6a3e1b2c0dbdb9c94c7089a82e94866e35507fddda959f6ea2f7 Jul 31 15:00:16.214343 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:16.214321 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m"] Jul 31 15:00:16.216460 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:00:16.216436 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8377efc7_9f86_4d3f_a88f_2697fc804f53.slice/crio-d7aa67a0b9c82b7a654f51dea887c490a18653e7d733faaa2f4e4aedd17d41f2 WatchSource:0}: Error finding container d7aa67a0b9c82b7a654f51dea887c490a18653e7d733faaa2f4e4aedd17d41f2: Status 404 returned error can't find the container with id d7aa67a0b9c82b7a654f51dea887c490a18653e7d733faaa2f4e4aedd17d41f2 Jul 31 15:00:16.899578 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:16.899542 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn" event={"ID":"70a7fb16-5c0d-4088-b0d0-ec91d5185db0","Type":"ContainerStarted","Data":"a4f31a8a5125ab2465737e48d4319f990517745ac0aac213815c7951b1a1aeaa"} Jul 31 15:00:16.900065 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:16.899700 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn" Jul 31 15:00:16.900065 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:16.899748 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn" event={"ID":"70a7fb16-5c0d-4088-b0d0-ec91d5185db0","Type":"ContainerStarted","Data":"6afa519c137d6a3e1b2c0dbdb9c94c7089a82e94866e35507fddda959f6ea2f7"} Jul 31 15:00:16.901095 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:16.901074 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m" event={"ID":"8377efc7-9f86-4d3f-a88f-2697fc804f53","Type":"ContainerStarted","Data":"65f2929e17e7ad56e249932e883e587dcbf811c172e1ba795992b9d86f79f7bd"} Jul 31 15:00:16.901200 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:16.901114 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m" Jul 31 15:00:16.901200 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:16.901130 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m" event={"ID":"8377efc7-9f86-4d3f-a88f-2697fc804f53","Type":"ContainerStarted","Data":"d7aa67a0b9c82b7a654f51dea887c490a18653e7d733faaa2f4e4aedd17d41f2"} Jul 31 15:00:16.915195 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:16.915152 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn" podStartSLOduration=1.9151384820000001 podStartE2EDuration="1.915138482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:00:16.912979805 +0000 UTC m=+2868.949802871" watchObservedRunningTime="2026-07-31 15:00:16.915138482 +0000 UTC m=+2868.951961549" Jul 31 15:00:16.928733 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:16.928688 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m" podStartSLOduration=1.928676123 podStartE2EDuration="1.928676123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:00:16.926938697 +0000 UTC m=+2868.963761768" watchObservedRunningTime="2026-07-31 15:00:16.928676123 +0000 UTC m=+2868.965499196" Jul 31 15:00:18.277420 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:18.277381 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:00:27.883938 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:27.883900 2577 patch_prober.go:28] interesting pod/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 31 15:00:27.884362 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:27.883966 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" podUID="89be0d85-23f1-4da0-ad33-feabc2518bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 31 15:00:27.907492 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:27.907468 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m" Jul 31 15:00:27.907612 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:27.907537 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn" Jul 31 15:00:27.955267 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:27.955231 2577 generic.go:356] "Generic (PLEG): container finished" podID="89be0d85-23f1-4da0-ad33-feabc2518bfa" containerID="1b8b7d42f57c7d05d5e7edf1213da5950498874b37f1ce9ab246cfe4dec50cde" exitCode=0 Jul 31 15:00:27.955428 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:27.955283 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" event={"ID":"89be0d85-23f1-4da0-ad33-feabc2518bfa","Type":"ContainerDied","Data":"1b8b7d42f57c7d05d5e7edf1213da5950498874b37f1ce9ab246cfe4dec50cde"} Jul 31 15:00:27.955680 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:27.955663 2577 scope.go:117] "RemoveContainer" containerID="1b8b7d42f57c7d05d5e7edf1213da5950498874b37f1ce9ab246cfe4dec50cde" Jul 31 15:00:27.956806 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:27.956776 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:00:27.956916 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:27.956861 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:00:27.956916 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:27.956899 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:00:28.961378 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:28.961344 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" event={"ID":"89be0d85-23f1-4da0-ad33-feabc2518bfa","Type":"ContainerStarted","Data":"e244ff0643f4775001933f78852de1ffa6f446c5a2e2917d9364bc011bab1c33"} Jul 31 15:00:29.522213 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:29.522178 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:29.522397 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:29.522373 2577 patch_prober.go:28] interesting pod/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 31 15:00:29.522500 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:29.522444 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" podUID="89be0d85-23f1-4da0-ad33-feabc2518bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 31 15:00:30.522946 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:30.522905 2577 patch_prober.go:28] interesting pod/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 31 15:00:30.523441 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:30.522981 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" podUID="89be0d85-23f1-4da0-ad33-feabc2518bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 31 15:00:31.522546 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:31.522505 2577 patch_prober.go:28] interesting pod/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 31 15:00:31.522790 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:31.522560 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" podUID="89be0d85-23f1-4da0-ad33-feabc2518bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 31 15:00:32.522686 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:32.522653 2577 patch_prober.go:28] interesting pod/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 31 15:00:32.523086 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:32.522700 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" podUID="89be0d85-23f1-4da0-ad33-feabc2518bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 31 15:00:33.522521 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:33.522485 2577 patch_prober.go:28] interesting pod/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 31 15:00:33.522697 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:33.522539 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" podUID="89be0d85-23f1-4da0-ad33-feabc2518bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 31 15:00:34.526052 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:34.526023 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:34.526440 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:34.526311 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:34.527029 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:34.527013 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:00:39.471889 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.471833 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz"] Jul 31 15:00:39.475028 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.475009 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.476627 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.476604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-data\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.476754 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.476659 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-workload-certs\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.476754 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.476725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-workload-socket\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.476754 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.476751 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-token\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.477073 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.476780 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-podinfo\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.477073 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.476796 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvx4h\" (UniqueName: \"kubernetes.io/projected/e8007a67-8653-434a-8a83-5de2b4170b5b-kube-api-access-fvx4h\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.477073 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.476825 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8007a67-8653-434a-8a83-5de2b4170b5b-istiod-ca-cert\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.477073 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.476931 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-credential-socket\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.477073 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.476971 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ef4b6d-openshift-default-dockercfg-sdk6h\"" Jul 31 15:00:39.477073 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.476978 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-envoy\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.486056 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.486028 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz"] Jul 31 15:00:39.577674 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.577640 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-workload-socket\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.577952 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.577757 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-token\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.577952 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.577796 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-podinfo\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.577952 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.577813 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvx4h\" (UniqueName: \"kubernetes.io/projected/e8007a67-8653-434a-8a83-5de2b4170b5b-kube-api-access-fvx4h\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.577952 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.577854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8007a67-8653-434a-8a83-5de2b4170b5b-istiod-ca-cert\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.577952 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.577887 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-credential-socket\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.577952 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.577925 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-envoy\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.578283 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.577991 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-data\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.578283 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.578038 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-workload-certs\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.578283 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.578146 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-workload-socket\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.578444 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.578334 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-credential-socket\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.578444 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.578429 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-data\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.578549 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.578494 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-workload-certs\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.578723 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.578699 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8007a67-8653-434a-8a83-5de2b4170b5b-istiod-ca-cert\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.580359 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.580333 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-envoy\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.580648 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.580630 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-podinfo\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.585388 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.585365 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-token\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.585690 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.585657 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvx4h\" (UniqueName: \"kubernetes.io/projected/e8007a67-8653-434a-8a83-5de2b4170b5b-kube-api-access-fvx4h\") pod \"e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.789575 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.789484 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:39.931181 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.931159 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz"] Jul 31 15:00:39.933563 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:00:39.933527 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8007a67_8653_434a_8a83_5de2b4170b5b.slice/crio-5d3a43e44698f2f65d92d646e94fd33c8f0d828acdb6b9696992b7cbca46e19c WatchSource:0}: Error finding container 5d3a43e44698f2f65d92d646e94fd33c8f0d828acdb6b9696992b7cbca46e19c: Status 404 returned error can't find the container with id 5d3a43e44698f2f65d92d646e94fd33c8f0d828acdb6b9696992b7cbca46e19c Jul 31 15:00:39.935783 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.935749 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:00:39.935918 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.935830 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:00:39.935989 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:39.935921 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:00:40.011407 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:40.011377 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" event={"ID":"e8007a67-8653-434a-8a83-5de2b4170b5b","Type":"ContainerStarted","Data":"3152f5f2bc44da4cbf713557ece9ef63ad75d7dc3c5af9abf07611dbc69d302c"} Jul 31 15:00:40.011534 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:40.011426 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" event={"ID":"e8007a67-8653-434a-8a83-5de2b4170b5b","Type":"ContainerStarted","Data":"5d3a43e44698f2f65d92d646e94fd33c8f0d828acdb6b9696992b7cbca46e19c"} Jul 31 15:00:40.032998 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:40.032917 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" podStartSLOduration=1.032895566 podStartE2EDuration="1.032895566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:00:40.030994173 +0000 UTC m=+2892.067817241" watchObservedRunningTime="2026-07-31 15:00:40.032895566 +0000 UTC m=+2892.069718635" Jul 31 15:00:40.789659 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:40.789623 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:40.794468 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:40.794443 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:41.015684 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:41.015658 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:41.016573 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:41.016556 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:45.583250 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:45.583216 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:00:55.091321 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:55.091281 2577 generic.go:356] "Generic (PLEG): container finished" podID="e8007a67-8653-434a-8a83-5de2b4170b5b" containerID="3152f5f2bc44da4cbf713557ece9ef63ad75d7dc3c5af9abf07611dbc69d302c" exitCode=0 Jul 31 15:00:55.091755 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:55.091357 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" event={"ID":"e8007a67-8653-434a-8a83-5de2b4170b5b","Type":"ContainerDied","Data":"3152f5f2bc44da4cbf713557ece9ef63ad75d7dc3c5af9abf07611dbc69d302c"} Jul 31 15:00:55.091755 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:55.091750 2577 scope.go:117] "RemoveContainer" containerID="3152f5f2bc44da4cbf713557ece9ef63ad75d7dc3c5af9abf07611dbc69d302c" Jul 31 15:00:55.092770 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:55.092741 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:00:55.092905 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:55.092829 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:00:55.092905 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:55.092893 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:00:56.016396 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:56.016346 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:56.097977 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:56.097939 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" event={"ID":"e8007a67-8653-434a-8a83-5de2b4170b5b","Type":"ContainerStarted","Data":"802b6f322505a3de88a6ae94fcca45b650bfec5890d3acec8ea805178ed871d9"} Jul 31 15:00:56.790026 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:56.789984 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:00:56.791076 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:56.791050 2577 patch_prober.go:28] interesting pod/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 31 15:00:56.791192 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:56.791119 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" podUID="e8007a67-8653-434a-8a83-5de2b4170b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 31 15:00:57.789959 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:57.789919 2577 patch_prober.go:28] interesting pod/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 31 15:00:57.790356 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:57.789983 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" podUID="e8007a67-8653-434a-8a83-5de2b4170b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 31 15:00:58.790254 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:58.790215 2577 patch_prober.go:28] interesting pod/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 31 15:00:58.790658 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:58.790285 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" podUID="e8007a67-8653-434a-8a83-5de2b4170b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 31 15:00:59.790266 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:59.790230 2577 patch_prober.go:28] interesting pod/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 31 15:00:59.790673 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:00:59.790286 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" podUID="e8007a67-8653-434a-8a83-5de2b4170b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 31 15:01:00.789860 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:01:00.789798 2577 patch_prober.go:28] interesting pod/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 31 15:01:00.790062 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:01:00.789894 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" podUID="e8007a67-8653-434a-8a83-5de2b4170b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 31 15:01:01.794427 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:01:01.794394 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:01:01.794944 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:01:01.794693 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:01:01.795293 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:01:01.795277 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:01:19.773369 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:01:19.773334 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:01:46.271874 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:01:46.271823 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:02:30.276542 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:30.276497 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:02:38.696196 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:38.696152 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m"] Jul 31 15:02:38.696669 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:38.696344 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m" podUID="8377efc7-9f86-4d3f-a88f-2697fc804f53" containerName="payload-processing" containerID="cri-o://65f2929e17e7ad56e249932e883e587dcbf811c172e1ba795992b9d86f79f7bd" gracePeriod=30 Jul 31 15:02:38.703162 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:38.703133 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn"] Jul 31 15:02:38.703408 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:38.703337 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn" podUID="70a7fb16-5c0d-4088-b0d0-ec91d5185db0" containerName="payload-pre-processing" containerID="cri-o://a4f31a8a5125ab2465737e48d4319f990517745ac0aac213815c7951b1a1aeaa" gracePeriod=30 Jul 31 15:02:38.974754 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:38.974725 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m" Jul 31 15:02:38.991750 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:38.991724 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn" Jul 31 15:02:39.124676 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.124645 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l77cr\" (UniqueName: \"kubernetes.io/projected/8377efc7-9f86-4d3f-a88f-2697fc804f53-kube-api-access-l77cr\") pod \"8377efc7-9f86-4d3f-a88f-2697fc804f53\" (UID: \"8377efc7-9f86-4d3f-a88f-2697fc804f53\") " Jul 31 15:02:39.124676 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.124681 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5l7d\" (UniqueName: \"kubernetes.io/projected/70a7fb16-5c0d-4088-b0d0-ec91d5185db0-kube-api-access-j5l7d\") pod \"70a7fb16-5c0d-4088-b0d0-ec91d5185db0\" (UID: \"70a7fb16-5c0d-4088-b0d0-ec91d5185db0\") " Jul 31 15:02:39.124898 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.124774 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8377efc7-9f86-4d3f-a88f-2697fc804f53-plugins-config-volume\") pod \"8377efc7-9f86-4d3f-a88f-2697fc804f53\" (UID: \"8377efc7-9f86-4d3f-a88f-2697fc804f53\") " Jul 31 15:02:39.124898 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.124799 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a7fb16-5c0d-4088-b0d0-ec91d5185db0-plugins-config-volume\") pod \"70a7fb16-5c0d-4088-b0d0-ec91d5185db0\" (UID: \"70a7fb16-5c0d-4088-b0d0-ec91d5185db0\") " Jul 31 15:02:39.125236 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.125195 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8377efc7-9f86-4d3f-a88f-2697fc804f53-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8377efc7-9f86-4d3f-a88f-2697fc804f53" (UID: "8377efc7-9f86-4d3f-a88f-2697fc804f53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:02:39.125236 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.125220 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a7fb16-5c0d-4088-b0d0-ec91d5185db0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70a7fb16-5c0d-4088-b0d0-ec91d5185db0" (UID: "70a7fb16-5c0d-4088-b0d0-ec91d5185db0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:02:39.126749 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.126726 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8377efc7-9f86-4d3f-a88f-2697fc804f53-kube-api-access-l77cr" (OuterVolumeSpecName: "kube-api-access-l77cr") pod "8377efc7-9f86-4d3f-a88f-2697fc804f53" (UID: "8377efc7-9f86-4d3f-a88f-2697fc804f53"). InnerVolumeSpecName "kube-api-access-l77cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:02:39.126817 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.126791 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a7fb16-5c0d-4088-b0d0-ec91d5185db0-kube-api-access-j5l7d" (OuterVolumeSpecName: "kube-api-access-j5l7d") pod "70a7fb16-5c0d-4088-b0d0-ec91d5185db0" (UID: "70a7fb16-5c0d-4088-b0d0-ec91d5185db0"). InnerVolumeSpecName "kube-api-access-j5l7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:02:39.225473 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.225408 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8377efc7-9f86-4d3f-a88f-2697fc804f53-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:02:39.225473 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.225430 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a7fb16-5c0d-4088-b0d0-ec91d5185db0-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:02:39.225473 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.225441 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l77cr\" (UniqueName: \"kubernetes.io/projected/8377efc7-9f86-4d3f-a88f-2697fc804f53-kube-api-access-l77cr\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:02:39.225473 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.225449 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5l7d\" (UniqueName: \"kubernetes.io/projected/70a7fb16-5c0d-4088-b0d0-ec91d5185db0-kube-api-access-j5l7d\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:02:39.561813 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.561721 2577 generic.go:356] "Generic (PLEG): container finished" podID="70a7fb16-5c0d-4088-b0d0-ec91d5185db0" containerID="a4f31a8a5125ab2465737e48d4319f990517745ac0aac213815c7951b1a1aeaa" exitCode=0 Jul 31 15:02:39.561813 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.561771 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn" Jul 31 15:02:39.562073 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.561809 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn" event={"ID":"70a7fb16-5c0d-4088-b0d0-ec91d5185db0","Type":"ContainerDied","Data":"a4f31a8a5125ab2465737e48d4319f990517745ac0aac213815c7951b1a1aeaa"} Jul 31 15:02:39.562073 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.561888 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn" event={"ID":"70a7fb16-5c0d-4088-b0d0-ec91d5185db0","Type":"ContainerDied","Data":"6afa519c137d6a3e1b2c0dbdb9c94c7089a82e94866e35507fddda959f6ea2f7"} Jul 31 15:02:39.562073 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.561919 2577 scope.go:117] "RemoveContainer" containerID="a4f31a8a5125ab2465737e48d4319f990517745ac0aac213815c7951b1a1aeaa" Jul 31 15:02:39.563080 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.563060 2577 generic.go:356] "Generic (PLEG): container finished" podID="8377efc7-9f86-4d3f-a88f-2697fc804f53" containerID="65f2929e17e7ad56e249932e883e587dcbf811c172e1ba795992b9d86f79f7bd" exitCode=0 Jul 31 15:02:39.563175 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.563107 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m" Jul 31 15:02:39.563218 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.563101 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m" event={"ID":"8377efc7-9f86-4d3f-a88f-2697fc804f53","Type":"ContainerDied","Data":"65f2929e17e7ad56e249932e883e587dcbf811c172e1ba795992b9d86f79f7bd"} Jul 31 15:02:39.563257 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.563227 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m" event={"ID":"8377efc7-9f86-4d3f-a88f-2697fc804f53","Type":"ContainerDied","Data":"d7aa67a0b9c82b7a654f51dea887c490a18653e7d733faaa2f4e4aedd17d41f2"} Jul 31 15:02:39.570162 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.570145 2577 scope.go:117] "RemoveContainer" containerID="a4f31a8a5125ab2465737e48d4319f990517745ac0aac213815c7951b1a1aeaa" Jul 31 15:02:39.570399 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:02:39.570378 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f31a8a5125ab2465737e48d4319f990517745ac0aac213815c7951b1a1aeaa\": container with ID starting with a4f31a8a5125ab2465737e48d4319f990517745ac0aac213815c7951b1a1aeaa not found: ID does not exist" containerID="a4f31a8a5125ab2465737e48d4319f990517745ac0aac213815c7951b1a1aeaa" Jul 31 15:02:39.570447 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.570406 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f31a8a5125ab2465737e48d4319f990517745ac0aac213815c7951b1a1aeaa"} err="failed to get container status \"a4f31a8a5125ab2465737e48d4319f990517745ac0aac213815c7951b1a1aeaa\": rpc error: code = NotFound desc = could not find container \"a4f31a8a5125ab2465737e48d4319f990517745ac0aac213815c7951b1a1aeaa\": container with ID starting with a4f31a8a5125ab2465737e48d4319f990517745ac0aac213815c7951b1a1aeaa not found: ID does not exist" Jul 31 15:02:39.570447 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.570422 2577 scope.go:117] "RemoveContainer" containerID="65f2929e17e7ad56e249932e883e587dcbf811c172e1ba795992b9d86f79f7bd" Jul 31 15:02:39.578080 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.578065 2577 scope.go:117] "RemoveContainer" containerID="65f2929e17e7ad56e249932e883e587dcbf811c172e1ba795992b9d86f79f7bd" Jul 31 15:02:39.578321 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:02:39.578304 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f2929e17e7ad56e249932e883e587dcbf811c172e1ba795992b9d86f79f7bd\": container with ID starting with 65f2929e17e7ad56e249932e883e587dcbf811c172e1ba795992b9d86f79f7bd not found: ID does not exist" containerID="65f2929e17e7ad56e249932e883e587dcbf811c172e1ba795992b9d86f79f7bd" Jul 31 15:02:39.578369 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.578327 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f2929e17e7ad56e249932e883e587dcbf811c172e1ba795992b9d86f79f7bd"} err="failed to get container status \"65f2929e17e7ad56e249932e883e587dcbf811c172e1ba795992b9d86f79f7bd\": rpc error: code = NotFound desc = could not find container \"65f2929e17e7ad56e249932e883e587dcbf811c172e1ba795992b9d86f79f7bd\": container with ID starting with 65f2929e17e7ad56e249932e883e587dcbf811c172e1ba795992b9d86f79f7bd not found: ID does not exist" Jul 31 15:02:39.586475 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.586454 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m"] Jul 31 15:02:39.588764 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.588744 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-55959d-58cc6b94dd-v6w8m"] Jul 31 15:02:39.598310 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.598291 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn"] Jul 31 15:02:39.607533 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:39.607514 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-55959d-7b87777776-xx4bn"] Jul 31 15:02:40.435690 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:40.435650 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70a7fb16-5c0d-4088-b0d0-ec91d5185db0" path="/var/lib/kubelet/pods/70a7fb16-5c0d-4088-b0d0-ec91d5185db0/volumes" Jul 31 15:02:40.436088 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:40.436014 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8377efc7-9f86-4d3f-a88f-2697fc804f53" path="/var/lib/kubelet/pods/8377efc7-9f86-4d3f-a88f-2697fc804f53/volumes" Jul 31 15:02:40.683642 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:40.683609 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:02:45.446751 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:45.446718 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m"] Jul 31 15:02:45.447173 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:45.446940 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" podUID="89be0d85-23f1-4da0-ad33-feabc2518bfa" containerName="istio-proxy" containerID="cri-o://e244ff0643f4775001933f78852de1ffa6f446c5a2e2917d9364bc011bab1c33" gracePeriod=30 Jul 31 15:02:49.528912 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:49.528872 2577 patch_prober.go:28] interesting pod/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 15:02:49.529264 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:49.528930 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" podUID="89be0d85-23f1-4da0-ad33-feabc2518bfa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 15:02:50.615296 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.615259 2577 generic.go:356] "Generic (PLEG): container finished" podID="89be0d85-23f1-4da0-ad33-feabc2518bfa" containerID="e244ff0643f4775001933f78852de1ffa6f446c5a2e2917d9364bc011bab1c33" exitCode=0 Jul 31 15:02:50.615725 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.615337 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" event={"ID":"89be0d85-23f1-4da0-ad33-feabc2518bfa","Type":"ContainerDied","Data":"e244ff0643f4775001933f78852de1ffa6f446c5a2e2917d9364bc011bab1c33"} Jul 31 15:02:50.615725 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.615398 2577 scope.go:117] "RemoveContainer" containerID="1b8b7d42f57c7d05d5e7edf1213da5950498874b37f1ce9ab246cfe4dec50cde" Jul 31 15:02:50.698293 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.698270 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:02:50.824507 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.824426 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-workload-certs\") pod \"89be0d85-23f1-4da0-ad33-feabc2518bfa\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " Jul 31 15:02:50.824507 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.824477 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-workload-socket\") pod \"89be0d85-23f1-4da0-ad33-feabc2518bfa\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " Jul 31 15:02:50.824731 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.824523 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-token\") pod \"89be0d85-23f1-4da0-ad33-feabc2518bfa\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " Jul 31 15:02:50.824731 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.824547 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-data\") pod \"89be0d85-23f1-4da0-ad33-feabc2518bfa\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " Jul 31 15:02:50.824731 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.824574 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-podinfo\") pod \"89be0d85-23f1-4da0-ad33-feabc2518bfa\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " Jul 31 15:02:50.824731 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.824604 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8gfk\" (UniqueName: \"kubernetes.io/projected/89be0d85-23f1-4da0-ad33-feabc2518bfa-kube-api-access-k8gfk\") pod \"89be0d85-23f1-4da0-ad33-feabc2518bfa\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " Jul 31 15:02:50.824731 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.824627 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-envoy\") pod \"89be0d85-23f1-4da0-ad33-feabc2518bfa\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " Jul 31 15:02:50.824731 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.824651 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89be0d85-23f1-4da0-ad33-feabc2518bfa-istiod-ca-cert\") pod \"89be0d85-23f1-4da0-ad33-feabc2518bfa\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " Jul 31 15:02:50.824731 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.824673 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-credential-socket\") pod \"89be0d85-23f1-4da0-ad33-feabc2518bfa\" (UID: \"89be0d85-23f1-4da0-ad33-feabc2518bfa\") " Jul 31 15:02:50.825122 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.824725 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "89be0d85-23f1-4da0-ad33-feabc2518bfa" (UID: "89be0d85-23f1-4da0-ad33-feabc2518bfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:02:50.825122 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.824869 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-data" (OuterVolumeSpecName: "istio-data") pod "89be0d85-23f1-4da0-ad33-feabc2518bfa" (UID: "89be0d85-23f1-4da0-ad33-feabc2518bfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:02:50.825122 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.824961 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "89be0d85-23f1-4da0-ad33-feabc2518bfa" (UID: "89be0d85-23f1-4da0-ad33-feabc2518bfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:02:50.825122 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.824981 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-data\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:02:50.825122 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.824996 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-workload-certs\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:02:50.825122 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.825081 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "89be0d85-23f1-4da0-ad33-feabc2518bfa" (UID: "89be0d85-23f1-4da0-ad33-feabc2518bfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:02:50.825366 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.825177 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89be0d85-23f1-4da0-ad33-feabc2518bfa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "89be0d85-23f1-4da0-ad33-feabc2518bfa" (UID: "89be0d85-23f1-4da0-ad33-feabc2518bfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:02:50.826672 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.826653 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-token" (OuterVolumeSpecName: "istio-token") pod "89be0d85-23f1-4da0-ad33-feabc2518bfa" (UID: "89be0d85-23f1-4da0-ad33-feabc2518bfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:02:50.826854 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.826820 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89be0d85-23f1-4da0-ad33-feabc2518bfa-kube-api-access-k8gfk" (OuterVolumeSpecName: "kube-api-access-k8gfk") pod "89be0d85-23f1-4da0-ad33-feabc2518bfa" (UID: "89be0d85-23f1-4da0-ad33-feabc2518bfa"). InnerVolumeSpecName "kube-api-access-k8gfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:02:50.827083 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.827065 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "89be0d85-23f1-4da0-ad33-feabc2518bfa" (UID: "89be0d85-23f1-4da0-ad33-feabc2518bfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:02:50.827124 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.827085 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "89be0d85-23f1-4da0-ad33-feabc2518bfa" (UID: "89be0d85-23f1-4da0-ad33-feabc2518bfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 15:02:50.925955 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.925931 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-token\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:02:50.925955 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.925951 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-podinfo\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:02:50.926100 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.925961 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8gfk\" (UniqueName: \"kubernetes.io/projected/89be0d85-23f1-4da0-ad33-feabc2518bfa-kube-api-access-k8gfk\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:02:50.926100 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.925970 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-istio-envoy\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:02:50.926100 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.925981 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89be0d85-23f1-4da0-ad33-feabc2518bfa-istiod-ca-cert\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:02:50.926100 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.925988 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-credential-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:02:50.926100 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:50.925996 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be0d85-23f1-4da0-ad33-feabc2518bfa-workload-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:02:51.621992 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:51.621964 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" Jul 31 15:02:51.622374 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:51.621963 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m" event={"ID":"89be0d85-23f1-4da0-ad33-feabc2518bfa","Type":"ContainerDied","Data":"64d29db13db81340b4cb18079f77997abaee012c371d14f8f3e529811474cdc1"} Jul 31 15:02:51.622374 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:51.622092 2577 scope.go:117] "RemoveContainer" containerID="e244ff0643f4775001933f78852de1ffa6f446c5a2e2917d9364bc011bab1c33" Jul 31 15:02:51.644768 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:51.644746 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m"] Jul 31 15:02:51.650644 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:51.650615 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-55959d-openshift-default-7bff4b675b-t8z9m"] Jul 31 15:02:52.429617 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:52.429580 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89be0d85-23f1-4da0-ad33-feabc2518bfa" path="/var/lib/kubelet/pods/89be0d85-23f1-4da0-ad33-feabc2518bfa/volumes" Jul 31 15:02:58.168468 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:02:58.168430 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:03:02.649545 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:02.649507 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz"] Jul 31 15:03:02.650190 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:02.649750 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" podUID="e8007a67-8653-434a-8a83-5de2b4170b5b" containerName="istio-proxy" containerID="cri-o://802b6f322505a3de88a6ae94fcca45b650bfec5890d3acec8ea805178ed871d9" gracePeriod=30 Jul 31 15:03:03.544003 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.543968 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck"] Jul 31 15:03:03.544279 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.544266 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8377efc7-9f86-4d3f-a88f-2697fc804f53" containerName="payload-processing" Jul 31 15:03:03.544328 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.544281 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8377efc7-9f86-4d3f-a88f-2697fc804f53" containerName="payload-processing" Jul 31 15:03:03.544328 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.544292 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a7fb16-5c0d-4088-b0d0-ec91d5185db0" containerName="payload-pre-processing" Jul 31 15:03:03.544328 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.544298 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a7fb16-5c0d-4088-b0d0-ec91d5185db0" containerName="payload-pre-processing" Jul 31 15:03:03.544328 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.544308 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89be0d85-23f1-4da0-ad33-feabc2518bfa" containerName="istio-proxy" Jul 31 15:03:03.544328 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.544313 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89be0d85-23f1-4da0-ad33-feabc2518bfa" containerName="istio-proxy" Jul 31 15:03:03.544328 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.544321 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89be0d85-23f1-4da0-ad33-feabc2518bfa" containerName="istio-proxy" Jul 31 15:03:03.544328 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.544327 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89be0d85-23f1-4da0-ad33-feabc2518bfa" containerName="istio-proxy" Jul 31 15:03:03.544542 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.544378 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89be0d85-23f1-4da0-ad33-feabc2518bfa" containerName="istio-proxy" Jul 31 15:03:03.544542 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.544385 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89be0d85-23f1-4da0-ad33-feabc2518bfa" containerName="istio-proxy" Jul 31 15:03:03.544542 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.544392 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8377efc7-9f86-4d3f-a88f-2697fc804f53" containerName="payload-processing" Jul 31 15:03:03.544542 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.544397 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="70a7fb16-5c0d-4088-b0d0-ec91d5185db0" containerName="payload-pre-processing" Jul 31 15:03:03.548886 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.548861 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.550939 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.550916 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-700618-openshift-default-dockercfg-8d24s\"" Jul 31 15:03:03.556960 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.556938 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck"] Jul 31 15:03:03.630739 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.630702 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66bfb387-27d3-485c-971f-3c27d346b63d-istio-token\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.630739 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.630749 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-workload-certs\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.631014 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.630785 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-credential-socket\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.631014 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.630936 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bfb387-27d3-485c-971f-3c27d346b63d-istiod-ca-cert\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.631014 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.631002 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-istio-data\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.631192 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.631032 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bxcb\" (UniqueName: \"kubernetes.io/projected/66bfb387-27d3-485c-971f-3c27d346b63d-kube-api-access-7bxcb\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.631192 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.631061 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-workload-socket\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.631192 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.631093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-istio-envoy\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.631192 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.631133 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bfb387-27d3-485c-971f-3c27d346b63d-istio-podinfo\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.731855 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.731815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-workload-certs\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.732343 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.731881 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-credential-socket\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.732343 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.731923 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bfb387-27d3-485c-971f-3c27d346b63d-istiod-ca-cert\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.732343 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.731962 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-istio-data\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.732343 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.731979 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bxcb\" (UniqueName: \"kubernetes.io/projected/66bfb387-27d3-485c-971f-3c27d346b63d-kube-api-access-7bxcb\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.732343 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.732097 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-workload-socket\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.732343 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.732152 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-istio-envoy\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.732343 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.732200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bfb387-27d3-485c-971f-3c27d346b63d-istio-podinfo\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.732343 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.732251 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-workload-certs\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.732343 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.732258 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66bfb387-27d3-485c-971f-3c27d346b63d-istio-token\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.732788 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.732386 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-credential-socket\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.732788 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.732455 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-workload-socket\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.732788 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.732596 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-istio-data\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.732788 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.732653 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bfb387-27d3-485c-971f-3c27d346b63d-istiod-ca-cert\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.734406 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.734387 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-istio-envoy\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.734674 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.734656 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bfb387-27d3-485c-971f-3c27d346b63d-istio-podinfo\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.739187 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.739153 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66bfb387-27d3-485c-971f-3c27d346b63d-istio-token\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.739293 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.739257 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bxcb\" (UniqueName: \"kubernetes.io/projected/66bfb387-27d3-485c-971f-3c27d346b63d-kube-api-access-7bxcb\") pod \"e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.859824 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.859733 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:03.989697 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.989671 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck"] Jul 31 15:03:03.991872 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:03:03.991829 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66bfb387_27d3_485c_971f_3c27d346b63d.slice/crio-fb84f59d805154772173abf9fc03019473c72ad9dbf3aa0e0ed96389c84dc566 WatchSource:0}: Error finding container fb84f59d805154772173abf9fc03019473c72ad9dbf3aa0e0ed96389c84dc566: Status 404 returned error can't find the container with id fb84f59d805154772173abf9fc03019473c72ad9dbf3aa0e0ed96389c84dc566 Jul 31 15:03:03.993932 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.993902 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:03:03.994007 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.993966 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:03:03.994007 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:03.993994 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:03:04.678326 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:04.678289 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" event={"ID":"66bfb387-27d3-485c-971f-3c27d346b63d","Type":"ContainerStarted","Data":"ea50df13998cb3887289657aa0409df5e53ee2fef9979495555ee29e71966bd0"} Jul 31 15:03:04.678505 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:04.678341 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" event={"ID":"66bfb387-27d3-485c-971f-3c27d346b63d","Type":"ContainerStarted","Data":"fb84f59d805154772173abf9fc03019473c72ad9dbf3aa0e0ed96389c84dc566"} Jul 31 15:03:04.697617 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:04.697575 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" podStartSLOduration=1.697560011 podStartE2EDuration="1.697560011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:03:04.696629503 +0000 UTC m=+3036.733452567" watchObservedRunningTime="2026-07-31 15:03:04.697560011 +0000 UTC m=+3036.734383078" Jul 31 15:03:04.859871 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:04.859831 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:05.865242 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:05.865212 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:06.686688 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:06.686648 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:06.687529 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:06.687507 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:07.900314 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.900293 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:03:07.965141 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.965064 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-token\") pod \"e8007a67-8653-434a-8a83-5de2b4170b5b\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " Jul 31 15:03:07.965141 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.965120 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-workload-certs\") pod \"e8007a67-8653-434a-8a83-5de2b4170b5b\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " Jul 31 15:03:07.965367 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.965144 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-podinfo\") pod \"e8007a67-8653-434a-8a83-5de2b4170b5b\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " Jul 31 15:03:07.965367 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.965166 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvx4h\" (UniqueName: \"kubernetes.io/projected/e8007a67-8653-434a-8a83-5de2b4170b5b-kube-api-access-fvx4h\") pod \"e8007a67-8653-434a-8a83-5de2b4170b5b\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " Jul 31 15:03:07.965367 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.965235 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-credential-socket\") pod \"e8007a67-8653-434a-8a83-5de2b4170b5b\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " Jul 31 15:03:07.965367 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.965268 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8007a67-8653-434a-8a83-5de2b4170b5b-istiod-ca-cert\") pod \"e8007a67-8653-434a-8a83-5de2b4170b5b\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " Jul 31 15:03:07.965367 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.965297 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-data\") pod \"e8007a67-8653-434a-8a83-5de2b4170b5b\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " Jul 31 15:03:07.965367 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.965320 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-workload-socket\") pod \"e8007a67-8653-434a-8a83-5de2b4170b5b\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " Jul 31 15:03:07.965367 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.965358 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-envoy\") pod \"e8007a67-8653-434a-8a83-5de2b4170b5b\" (UID: \"e8007a67-8653-434a-8a83-5de2b4170b5b\") " Jul 31 15:03:07.965753 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.965414 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e8007a67-8653-434a-8a83-5de2b4170b5b" (UID: "e8007a67-8653-434a-8a83-5de2b4170b5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:03:07.965753 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.965660 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-workload-certs\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:03:07.965753 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.965720 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-data" (OuterVolumeSpecName: "istio-data") pod "e8007a67-8653-434a-8a83-5de2b4170b5b" (UID: "e8007a67-8653-434a-8a83-5de2b4170b5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:03:07.965950 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.965750 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8007a67-8653-434a-8a83-5de2b4170b5b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e8007a67-8653-434a-8a83-5de2b4170b5b" (UID: "e8007a67-8653-434a-8a83-5de2b4170b5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:03:07.965950 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.965882 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e8007a67-8653-434a-8a83-5de2b4170b5b" (UID: "e8007a67-8653-434a-8a83-5de2b4170b5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:03:07.966065 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.965994 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e8007a67-8653-434a-8a83-5de2b4170b5b" (UID: "e8007a67-8653-434a-8a83-5de2b4170b5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:03:07.967488 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.967457 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e8007a67-8653-434a-8a83-5de2b4170b5b" (UID: "e8007a67-8653-434a-8a83-5de2b4170b5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:03:07.967619 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.967601 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-token" (OuterVolumeSpecName: "istio-token") pod "e8007a67-8653-434a-8a83-5de2b4170b5b" (UID: "e8007a67-8653-434a-8a83-5de2b4170b5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:03:07.967712 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.967697 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e8007a67-8653-434a-8a83-5de2b4170b5b" (UID: "e8007a67-8653-434a-8a83-5de2b4170b5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 15:03:07.967888 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:07.967871 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8007a67-8653-434a-8a83-5de2b4170b5b-kube-api-access-fvx4h" (OuterVolumeSpecName: "kube-api-access-fvx4h") pod "e8007a67-8653-434a-8a83-5de2b4170b5b" (UID: "e8007a67-8653-434a-8a83-5de2b4170b5b"). InnerVolumeSpecName "kube-api-access-fvx4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:03:08.066651 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.066616 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-credential-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:03:08.066651 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.066644 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8007a67-8653-434a-8a83-5de2b4170b5b-istiod-ca-cert\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:03:08.066651 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.066654 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-data\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:03:08.066958 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.066663 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-workload-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:03:08.066958 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.066671 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-envoy\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:03:08.066958 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.066679 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-token\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:03:08.066958 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.066687 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8007a67-8653-434a-8a83-5de2b4170b5b-istio-podinfo\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:03:08.066958 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.066695 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvx4h\" (UniqueName: \"kubernetes.io/projected/e8007a67-8653-434a-8a83-5de2b4170b5b-kube-api-access-fvx4h\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:03:08.297233 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.297154 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh"] Jul 31 15:03:08.297474 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.297462 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8007a67-8653-434a-8a83-5de2b4170b5b" containerName="istio-proxy" Jul 31 15:03:08.297520 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.297476 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8007a67-8653-434a-8a83-5de2b4170b5b" containerName="istio-proxy" Jul 31 15:03:08.297555 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.297545 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8007a67-8653-434a-8a83-5de2b4170b5b" containerName="istio-proxy" Jul 31 15:03:08.297555 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.297553 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8007a67-8653-434a-8a83-5de2b4170b5b" containerName="istio-proxy" Jul 31 15:03:08.300575 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.300559 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh" Jul 31 15:03:08.302923 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.302897 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-700618\"" Jul 31 15:03:08.303042 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.303007 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-700618-dockercfg-nz8ls\"" Jul 31 15:03:08.309165 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.309146 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh"] Jul 31 15:03:08.323890 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.323865 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj"] Jul 31 15:03:08.324147 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.324135 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8007a67-8653-434a-8a83-5de2b4170b5b" containerName="istio-proxy" Jul 31 15:03:08.324147 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.324148 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8007a67-8653-434a-8a83-5de2b4170b5b" containerName="istio-proxy" Jul 31 15:03:08.326904 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.326888 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj" Jul 31 15:03:08.334403 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.334380 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj"] Jul 31 15:03:08.369666 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.369642 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jprbz\" (UniqueName: \"kubernetes.io/projected/4466ec0b-f347-4fea-8e90-19e808a787ab-kube-api-access-jprbz\") pod \"payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh\" (UID: \"4466ec0b-f347-4fea-8e90-19e808a787ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh" Jul 31 15:03:08.369798 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.369688 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4466ec0b-f347-4fea-8e90-19e808a787ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh\" (UID: \"4466ec0b-f347-4fea-8e90-19e808a787ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh" Jul 31 15:03:08.369798 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.369713 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2n8x\" (UniqueName: \"kubernetes.io/projected/94d31d10-7b9e-4668-8fcb-1a86406b079d-kube-api-access-d2n8x\") pod \"payload-processing-e2e-api-a-700618-5d448c948d-v68sj\" (UID: \"94d31d10-7b9e-4668-8fcb-1a86406b079d\") " pod="openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj" Jul 31 15:03:08.369798 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.369787 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d31d10-7b9e-4668-8fcb-1a86406b079d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-700618-5d448c948d-v68sj\" (UID: \"94d31d10-7b9e-4668-8fcb-1a86406b079d\") " pod="openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj" Jul 31 15:03:08.471170 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.471135 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jprbz\" (UniqueName: \"kubernetes.io/projected/4466ec0b-f347-4fea-8e90-19e808a787ab-kube-api-access-jprbz\") pod \"payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh\" (UID: \"4466ec0b-f347-4fea-8e90-19e808a787ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh" Jul 31 15:03:08.471289 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.471175 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4466ec0b-f347-4fea-8e90-19e808a787ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh\" (UID: \"4466ec0b-f347-4fea-8e90-19e808a787ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh" Jul 31 15:03:08.471289 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.471198 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2n8x\" (UniqueName: \"kubernetes.io/projected/94d31d10-7b9e-4668-8fcb-1a86406b079d-kube-api-access-d2n8x\") pod \"payload-processing-e2e-api-a-700618-5d448c948d-v68sj\" (UID: \"94d31d10-7b9e-4668-8fcb-1a86406b079d\") " pod="openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj" Jul 31 15:03:08.471289 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.471260 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d31d10-7b9e-4668-8fcb-1a86406b079d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-700618-5d448c948d-v68sj\" (UID: \"94d31d10-7b9e-4668-8fcb-1a86406b079d\") " pod="openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj" Jul 31 15:03:08.471853 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.471827 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4466ec0b-f347-4fea-8e90-19e808a787ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh\" (UID: \"4466ec0b-f347-4fea-8e90-19e808a787ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh" Jul 31 15:03:08.471939 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.471923 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d31d10-7b9e-4668-8fcb-1a86406b079d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-700618-5d448c948d-v68sj\" (UID: \"94d31d10-7b9e-4668-8fcb-1a86406b079d\") " pod="openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj" Jul 31 15:03:08.478711 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.478690 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jprbz\" (UniqueName: \"kubernetes.io/projected/4466ec0b-f347-4fea-8e90-19e808a787ab-kube-api-access-jprbz\") pod \"payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh\" (UID: \"4466ec0b-f347-4fea-8e90-19e808a787ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh" Jul 31 15:03:08.478862 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.478817 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2n8x\" (UniqueName: \"kubernetes.io/projected/94d31d10-7b9e-4668-8fcb-1a86406b079d-kube-api-access-d2n8x\") pod \"payload-processing-e2e-api-a-700618-5d448c948d-v68sj\" (UID: \"94d31d10-7b9e-4668-8fcb-1a86406b079d\") " pod="openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj" Jul 31 15:03:08.611555 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.611474 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh" Jul 31 15:03:08.637468 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.637436 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj" Jul 31 15:03:08.699855 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.699313 2577 generic.go:356] "Generic (PLEG): container finished" podID="e8007a67-8653-434a-8a83-5de2b4170b5b" containerID="802b6f322505a3de88a6ae94fcca45b650bfec5890d3acec8ea805178ed871d9" exitCode=0 Jul 31 15:03:08.699855 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.699417 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" Jul 31 15:03:08.699855 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.699456 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" event={"ID":"e8007a67-8653-434a-8a83-5de2b4170b5b","Type":"ContainerDied","Data":"802b6f322505a3de88a6ae94fcca45b650bfec5890d3acec8ea805178ed871d9"} Jul 31 15:03:08.699855 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.699510 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz" event={"ID":"e8007a67-8653-434a-8a83-5de2b4170b5b","Type":"ContainerDied","Data":"5d3a43e44698f2f65d92d646e94fd33c8f0d828acdb6b9696992b7cbca46e19c"} Jul 31 15:03:08.699855 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.699543 2577 scope.go:117] "RemoveContainer" containerID="802b6f322505a3de88a6ae94fcca45b650bfec5890d3acec8ea805178ed871d9" Jul 31 15:03:08.712540 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.712516 2577 scope.go:117] "RemoveContainer" containerID="3152f5f2bc44da4cbf713557ece9ef63ad75d7dc3c5af9abf07611dbc69d302c" Jul 31 15:03:08.720752 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.720701 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz"] Jul 31 15:03:08.722081 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.722053 2577 scope.go:117] "RemoveContainer" containerID="802b6f322505a3de88a6ae94fcca45b650bfec5890d3acec8ea805178ed871d9" Jul 31 15:03:08.722383 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:03:08.722362 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"802b6f322505a3de88a6ae94fcca45b650bfec5890d3acec8ea805178ed871d9\": container with ID starting with 802b6f322505a3de88a6ae94fcca45b650bfec5890d3acec8ea805178ed871d9 not found: ID does not exist" containerID="802b6f322505a3de88a6ae94fcca45b650bfec5890d3acec8ea805178ed871d9" Jul 31 15:03:08.722454 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.722399 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"802b6f322505a3de88a6ae94fcca45b650bfec5890d3acec8ea805178ed871d9"} err="failed to get container status \"802b6f322505a3de88a6ae94fcca45b650bfec5890d3acec8ea805178ed871d9\": rpc error: code = NotFound desc = could not find container \"802b6f322505a3de88a6ae94fcca45b650bfec5890d3acec8ea805178ed871d9\": container with ID starting with 802b6f322505a3de88a6ae94fcca45b650bfec5890d3acec8ea805178ed871d9 not found: ID does not exist" Jul 31 15:03:08.722454 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.722419 2577 scope.go:117] "RemoveContainer" containerID="3152f5f2bc44da4cbf713557ece9ef63ad75d7dc3c5af9abf07611dbc69d302c" Jul 31 15:03:08.722772 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:03:08.722703 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3152f5f2bc44da4cbf713557ece9ef63ad75d7dc3c5af9abf07611dbc69d302c\": container with ID starting with 3152f5f2bc44da4cbf713557ece9ef63ad75d7dc3c5af9abf07611dbc69d302c not found: ID does not exist" containerID="3152f5f2bc44da4cbf713557ece9ef63ad75d7dc3c5af9abf07611dbc69d302c" Jul 31 15:03:08.722772 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.722738 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3152f5f2bc44da4cbf713557ece9ef63ad75d7dc3c5af9abf07611dbc69d302c"} err="failed to get container status \"3152f5f2bc44da4cbf713557ece9ef63ad75d7dc3c5af9abf07611dbc69d302c\": rpc error: code = NotFound desc = could not find container \"3152f5f2bc44da4cbf713557ece9ef63ad75d7dc3c5af9abf07611dbc69d302c\": container with ID starting with 3152f5f2bc44da4cbf713557ece9ef63ad75d7dc3c5af9abf07611dbc69d302c not found: ID does not exist" Jul 31 15:03:08.726191 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.726132 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ef4b6d-openshift-default-7f95b865c8-2b9rz"] Jul 31 15:03:08.758778 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.758751 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh"] Jul 31 15:03:08.762088 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:03:08.762057 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4466ec0b_f347_4fea_8e90_19e808a787ab.slice/crio-d9e2e312c1f2e25626f93cfaa5c62aa56d220a76518e359b0f8ed4b1eabce72a WatchSource:0}: Error finding container d9e2e312c1f2e25626f93cfaa5c62aa56d220a76518e359b0f8ed4b1eabce72a: Status 404 returned error can't find the container with id d9e2e312c1f2e25626f93cfaa5c62aa56d220a76518e359b0f8ed4b1eabce72a Jul 31 15:03:08.785815 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:08.785788 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj"] Jul 31 15:03:08.794063 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:03:08.794040 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d31d10_7b9e_4668_8fcb_1a86406b079d.slice/crio-a5cbf0cf2e079eb4208132ef203a983571e85fa6ad834403bace094e06a1c087 WatchSource:0}: Error finding container a5cbf0cf2e079eb4208132ef203a983571e85fa6ad834403bace094e06a1c087: Status 404 returned error can't find the container with id a5cbf0cf2e079eb4208132ef203a983571e85fa6ad834403bace094e06a1c087 Jul 31 15:03:09.704907 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:09.704868 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj" event={"ID":"94d31d10-7b9e-4668-8fcb-1a86406b079d","Type":"ContainerStarted","Data":"1ec557a3ed0ca0240f46b93e981cbe54aeaaaeebe923ba9fcc0364cc5ff6e3ef"} Jul 31 15:03:09.705368 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:09.704932 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj" event={"ID":"94d31d10-7b9e-4668-8fcb-1a86406b079d","Type":"ContainerStarted","Data":"a5cbf0cf2e079eb4208132ef203a983571e85fa6ad834403bace094e06a1c087"} Jul 31 15:03:09.705368 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:09.704987 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj" Jul 31 15:03:09.706417 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:09.706392 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh" event={"ID":"4466ec0b-f347-4fea-8e90-19e808a787ab","Type":"ContainerStarted","Data":"df15352eb37fe0649896c77e39702c162430ccf3c7f001c1c83656722ea3e2a1"} Jul 31 15:03:09.706552 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:09.706437 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh" event={"ID":"4466ec0b-f347-4fea-8e90-19e808a787ab","Type":"ContainerStarted","Data":"d9e2e312c1f2e25626f93cfaa5c62aa56d220a76518e359b0f8ed4b1eabce72a"} Jul 31 15:03:09.706552 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:09.706502 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh" Jul 31 15:03:09.719357 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:09.719308 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj" podStartSLOduration=1.719293327 podStartE2EDuration="1.719293327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:03:09.718721024 +0000 UTC m=+3041.755544094" watchObservedRunningTime="2026-07-31 15:03:09.719293327 +0000 UTC m=+3041.756116394" Jul 31 15:03:09.733604 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:09.733562 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh" podStartSLOduration=1.7335495239999998 podStartE2EDuration="1.733549524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:03:09.73200584 +0000 UTC m=+3041.768828906" watchObservedRunningTime="2026-07-31 15:03:09.733549524 +0000 UTC m=+3041.770372590" Jul 31 15:03:10.275049 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:10.275014 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:03:10.430011 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:10.429978 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8007a67-8653-434a-8a83-5de2b4170b5b" path="/var/lib/kubelet/pods/e8007a67-8653-434a-8a83-5de2b4170b5b/volumes" Jul 31 15:03:20.714084 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:20.714007 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj" Jul 31 15:03:20.714084 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:20.714082 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh" Jul 31 15:03:20.768205 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:20.768175 2577 generic.go:356] "Generic (PLEG): container finished" podID="66bfb387-27d3-485c-971f-3c27d346b63d" containerID="ea50df13998cb3887289657aa0409df5e53ee2fef9979495555ee29e71966bd0" exitCode=0 Jul 31 15:03:20.768360 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:20.768235 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" event={"ID":"66bfb387-27d3-485c-971f-3c27d346b63d","Type":"ContainerDied","Data":"ea50df13998cb3887289657aa0409df5e53ee2fef9979495555ee29e71966bd0"} Jul 31 15:03:20.768583 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:20.768570 2577 scope.go:117] "RemoveContainer" containerID="ea50df13998cb3887289657aa0409df5e53ee2fef9979495555ee29e71966bd0" Jul 31 15:03:20.769641 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:20.769606 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:03:20.769708 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:20.769692 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:03:20.769752 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:20.769733 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:03:21.686863 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:21.686806 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:21.774486 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:21.774451 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" event={"ID":"66bfb387-27d3-485c-971f-3c27d346b63d","Type":"ContainerStarted","Data":"a06fb64c3d2a53de0bad896cc70a9a92692b6e1b8f706231de246f2e146cf185"} Jul 31 15:03:21.860944 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:21.860907 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:21.862142 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:21.862116 2577 patch_prober.go:28] interesting pod/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 31 15:03:21.862267 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:21.862165 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" podUID="66bfb387-27d3-485c-971f-3c27d346b63d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 31 15:03:22.861134 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:22.861091 2577 patch_prober.go:28] interesting pod/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 31 15:03:22.861626 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:22.861158 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" podUID="66bfb387-27d3-485c-971f-3c27d346b63d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 31 15:03:23.860630 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:23.860594 2577 patch_prober.go:28] interesting pod/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 31 15:03:23.860811 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:23.860644 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" podUID="66bfb387-27d3-485c-971f-3c27d346b63d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 31 15:03:24.860370 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:24.860328 2577 patch_prober.go:28] interesting pod/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 31 15:03:24.860777 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:24.860393 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" podUID="66bfb387-27d3-485c-971f-3c27d346b63d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 31 15:03:25.860451 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:25.860410 2577 patch_prober.go:28] interesting pod/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 31 15:03:25.860895 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:25.860469 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" podUID="66bfb387-27d3-485c-971f-3c27d346b63d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 31 15:03:26.864477 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:26.864443 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:26.864869 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:26.864828 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:26.865548 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:26.865530 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:03:32.378959 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.378925 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h"] Jul 31 15:03:32.384137 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.384112 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.389484 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.386072 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2eeeef-openshift-default-dockercfg-qcm95\"" Jul 31 15:03:32.394236 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.394152 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h"] Jul 31 15:03:32.513599 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.513563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-credential-socket\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.513790 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.513623 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-envoy\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.513790 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.513643 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qnk4\" (UniqueName: \"kubernetes.io/projected/d7dfc061-287f-40fd-82db-f50c8b698e12-kube-api-access-9qnk4\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.513790 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.513773 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-workload-certs\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.513989 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.513829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-workload-socket\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.513989 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.513880 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-podinfo\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.513989 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.513936 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-data\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.513989 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.513957 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-token\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.513989 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.513976 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7dfc061-287f-40fd-82db-f50c8b698e12-istiod-ca-cert\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.615309 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.615281 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-envoy\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.615448 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.615314 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qnk4\" (UniqueName: \"kubernetes.io/projected/d7dfc061-287f-40fd-82db-f50c8b698e12-kube-api-access-9qnk4\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.615448 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.615347 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-workload-certs\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.615448 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.615386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-workload-socket\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.615448 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.615431 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-podinfo\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.615656 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.615576 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-data\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.615656 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.615627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-token\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.615766 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.615672 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7dfc061-287f-40fd-82db-f50c8b698e12-istiod-ca-cert\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.615766 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.615713 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-credential-socket\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.615907 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.615788 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-workload-certs\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.615907 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.615862 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-workload-socket\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.616022 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.615942 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-data\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.616107 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.616089 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-credential-socket\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.616418 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.616401 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7dfc061-287f-40fd-82db-f50c8b698e12-istiod-ca-cert\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.617700 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.617678 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-envoy\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.617944 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.617927 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-podinfo\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.622881 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.622857 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-token\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.623047 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.623032 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qnk4\" (UniqueName: \"kubernetes.io/projected/d7dfc061-287f-40fd-82db-f50c8b698e12-kube-api-access-9qnk4\") pod \"e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.695486 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.695416 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:32.844252 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.844225 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h"] Jul 31 15:03:32.847007 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:03:32.846981 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7dfc061_287f_40fd_82db_f50c8b698e12.slice/crio-15f5e9d417b382755f99292cb564ef6ceb4240c95fcf68ba22af6df0c575e872 WatchSource:0}: Error finding container 15f5e9d417b382755f99292cb564ef6ceb4240c95fcf68ba22af6df0c575e872: Status 404 returned error can't find the container with id 15f5e9d417b382755f99292cb564ef6ceb4240c95fcf68ba22af6df0c575e872 Jul 31 15:03:32.849113 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.849082 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:03:32.849218 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.849142 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:03:32.849218 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:32.849170 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:03:33.826896 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:33.826866 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" event={"ID":"d7dfc061-287f-40fd-82db-f50c8b698e12","Type":"ContainerStarted","Data":"2aa5772b6bdda49d84a77ff3b270c67cf8fa18ff54f45b755816b49bd6f316ea"} Jul 31 15:03:33.827351 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:33.826914 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" event={"ID":"d7dfc061-287f-40fd-82db-f50c8b698e12","Type":"ContainerStarted","Data":"15f5e9d417b382755f99292cb564ef6ceb4240c95fcf68ba22af6df0c575e872"} Jul 31 15:03:33.843992 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:33.843762 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" podStartSLOduration=1.843743948 podStartE2EDuration="1.843743948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:03:33.843329844 +0000 UTC m=+3065.880152913" watchObservedRunningTime="2026-07-31 15:03:33.843743948 +0000 UTC m=+3065.880567014" Jul 31 15:03:34.695820 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:34.695784 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:34.700217 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:34.700192 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:34.831787 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:34.831759 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:34.832552 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:34.832536 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:37.628602 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:37.628568 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8"] Jul 31 15:03:37.631979 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:37.631962 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8" Jul 31 15:03:37.633979 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:37.633956 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2eeeef\"" Jul 31 15:03:37.640695 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:37.640665 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8"] Jul 31 15:03:37.761738 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:37.761700 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn6fl\" (UniqueName: \"kubernetes.io/projected/cfa4b272-5198-4716-b4ef-5c9240606f0e-kube-api-access-tn6fl\") pod \"payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8\" (UID: \"cfa4b272-5198-4716-b4ef-5c9240606f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8" Jul 31 15:03:37.761957 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:37.761811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfa4b272-5198-4716-b4ef-5c9240606f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8\" (UID: \"cfa4b272-5198-4716-b4ef-5c9240606f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8" Jul 31 15:03:37.862658 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:37.862626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfa4b272-5198-4716-b4ef-5c9240606f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8\" (UID: \"cfa4b272-5198-4716-b4ef-5c9240606f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8" Jul 31 15:03:37.862815 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:37.862687 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn6fl\" (UniqueName: \"kubernetes.io/projected/cfa4b272-5198-4716-b4ef-5c9240606f0e-kube-api-access-tn6fl\") pod \"payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8\" (UID: \"cfa4b272-5198-4716-b4ef-5c9240606f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8" Jul 31 15:03:37.863324 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:37.863304 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfa4b272-5198-4716-b4ef-5c9240606f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8\" (UID: \"cfa4b272-5198-4716-b4ef-5c9240606f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8" Jul 31 15:03:37.869970 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:37.869945 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn6fl\" (UniqueName: \"kubernetes.io/projected/cfa4b272-5198-4716-b4ef-5c9240606f0e-kube-api-access-tn6fl\") pod \"payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8\" (UID: \"cfa4b272-5198-4716-b4ef-5c9240606f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8" Jul 31 15:03:37.943926 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:37.943834 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8" Jul 31 15:03:38.070764 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:38.070733 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8"] Jul 31 15:03:38.073948 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:03:38.073917 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa4b272_5198_4716_b4ef_5c9240606f0e.slice/crio-466bb52590ff4505a97c1c73298d5eb648333f3fd99eb00f3b036a15a3635423 WatchSource:0}: Error finding container 466bb52590ff4505a97c1c73298d5eb648333f3fd99eb00f3b036a15a3635423: Status 404 returned error can't find the container with id 466bb52590ff4505a97c1c73298d5eb648333f3fd99eb00f3b036a15a3635423 Jul 31 15:03:38.850493 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:38.850451 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8" event={"ID":"cfa4b272-5198-4716-b4ef-5c9240606f0e","Type":"ContainerStarted","Data":"eb43d8bf881920f0fbb7b51527377db882d3f45310ab41c785779545648c2a3b"} Jul 31 15:03:38.850493 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:38.850500 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8" event={"ID":"cfa4b272-5198-4716-b4ef-5c9240606f0e","Type":"ContainerStarted","Data":"466bb52590ff4505a97c1c73298d5eb648333f3fd99eb00f3b036a15a3635423"} Jul 31 15:03:38.850980 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:38.850531 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8" Jul 31 15:03:38.864458 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:38.864408 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8" podStartSLOduration=1.8643905809999999 podStartE2EDuration="1.864390581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:03:38.864091919 +0000 UTC m=+3070.900914986" watchObservedRunningTime="2026-07-31 15:03:38.864390581 +0000 UTC m=+3070.901213646" Jul 31 15:03:40.978274 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:40.978238 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:03:49.857478 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:49.857445 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8" Jul 31 15:03:50.711718 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:50.711681 2577 patch_prober.go:28] interesting pod/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": read tcp 10.133.0.2:46772->10.133.0.45:15021: read: connection reset by peer" start-of-body= Jul 31 15:03:50.712013 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:50.711753 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" podUID="d7dfc061-287f-40fd-82db-f50c8b698e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": read tcp 10.133.0.2:46772->10.133.0.45:15021: read: connection reset by peer" Jul 31 15:03:50.906023 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:50.905990 2577 generic.go:356] "Generic (PLEG): container finished" podID="d7dfc061-287f-40fd-82db-f50c8b698e12" containerID="2aa5772b6bdda49d84a77ff3b270c67cf8fa18ff54f45b755816b49bd6f316ea" exitCode=0 Jul 31 15:03:50.906483 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:50.906068 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" event={"ID":"d7dfc061-287f-40fd-82db-f50c8b698e12","Type":"ContainerDied","Data":"2aa5772b6bdda49d84a77ff3b270c67cf8fa18ff54f45b755816b49bd6f316ea"} Jul 31 15:03:50.906580 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:50.906560 2577 scope.go:117] "RemoveContainer" containerID="2aa5772b6bdda49d84a77ff3b270c67cf8fa18ff54f45b755816b49bd6f316ea" Jul 31 15:03:50.907615 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:50.907588 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:03:50.907682 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:50.907660 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:03:50.907725 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:50.907699 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:03:51.912433 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:51.912393 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" event={"ID":"d7dfc061-287f-40fd-82db-f50c8b698e12","Type":"ContainerStarted","Data":"64e2d3e675a669d612fb85f6195679aa2af4cd302a4787eaecc2cd5e413ee2fa"} Jul 31 15:03:52.695822 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:52.695792 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:52.696006 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:52.695974 2577 patch_prober.go:28] interesting pod/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 31 15:03:52.696051 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:52.696021 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" podUID="d7dfc061-287f-40fd-82db-f50c8b698e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 31 15:03:53.696071 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:53.696033 2577 patch_prober.go:28] interesting pod/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 31 15:03:53.696498 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:53.696091 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" podUID="d7dfc061-287f-40fd-82db-f50c8b698e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 31 15:03:54.696304 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:54.696267 2577 patch_prober.go:28] interesting pod/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 31 15:03:54.696703 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:54.696338 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" podUID="d7dfc061-287f-40fd-82db-f50c8b698e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 31 15:03:55.695986 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:55.695940 2577 patch_prober.go:28] interesting pod/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 31 15:03:55.696171 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:55.696031 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" podUID="d7dfc061-287f-40fd-82db-f50c8b698e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 31 15:03:56.696479 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:56.696446 2577 patch_prober.go:28] interesting pod/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 31 15:03:56.696887 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:56.696512 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" podUID="d7dfc061-287f-40fd-82db-f50c8b698e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 31 15:03:57.700066 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:57.700034 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:57.700473 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:57.700381 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:03:57.701034 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:03:57.701015 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:04:19.328647 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:19.328615 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj"] Jul 31 15:04:19.329328 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:19.328860 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj" podUID="94d31d10-7b9e-4668-8fcb-1a86406b079d" containerName="payload-processing" containerID="cri-o://1ec557a3ed0ca0240f46b93e981cbe54aeaaaeebe923ba9fcc0364cc5ff6e3ef" gracePeriod=30 Jul 31 15:04:19.337704 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:19.337449 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh"] Jul 31 15:04:19.337704 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:19.337672 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh" podUID="4466ec0b-f347-4fea-8e90-19e808a787ab" containerName="payload-pre-processing" containerID="cri-o://df15352eb37fe0649896c77e39702c162430ccf3c7f001c1c83656722ea3e2a1" gracePeriod=30 Jul 31 15:04:19.611620 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:19.611591 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj" Jul 31 15:04:19.614376 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:19.614359 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh" Jul 31 15:04:19.719695 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:19.719663 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d31d10-7b9e-4668-8fcb-1a86406b079d-plugins-config-volume\") pod \"94d31d10-7b9e-4668-8fcb-1a86406b079d\" (UID: \"94d31d10-7b9e-4668-8fcb-1a86406b079d\") " Jul 31 15:04:19.719871 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:19.719723 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jprbz\" (UniqueName: \"kubernetes.io/projected/4466ec0b-f347-4fea-8e90-19e808a787ab-kube-api-access-jprbz\") pod \"4466ec0b-f347-4fea-8e90-19e808a787ab\" (UID: \"4466ec0b-f347-4fea-8e90-19e808a787ab\") " Jul 31 15:04:19.719871 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:19.719797 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4466ec0b-f347-4fea-8e90-19e808a787ab-plugins-config-volume\") pod \"4466ec0b-f347-4fea-8e90-19e808a787ab\" (UID: \"4466ec0b-f347-4fea-8e90-19e808a787ab\") " Jul 31 15:04:19.719871 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:19.719829 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2n8x\" (UniqueName: \"kubernetes.io/projected/94d31d10-7b9e-4668-8fcb-1a86406b079d-kube-api-access-d2n8x\") pod \"94d31d10-7b9e-4668-8fcb-1a86406b079d\" (UID: \"94d31d10-7b9e-4668-8fcb-1a86406b079d\") " Jul 31 15:04:19.720100 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:19.720074 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d31d10-7b9e-4668-8fcb-1a86406b079d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "94d31d10-7b9e-4668-8fcb-1a86406b079d" (UID: "94d31d10-7b9e-4668-8fcb-1a86406b079d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:04:19.720175 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:19.720157 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d31d10-7b9e-4668-8fcb-1a86406b079d-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:19.720214 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:19.720192 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4466ec0b-f347-4fea-8e90-19e808a787ab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4466ec0b-f347-4fea-8e90-19e808a787ab" (UID: "4466ec0b-f347-4fea-8e90-19e808a787ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:04:19.721799 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:19.721777 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4466ec0b-f347-4fea-8e90-19e808a787ab-kube-api-access-jprbz" (OuterVolumeSpecName: "kube-api-access-jprbz") pod "4466ec0b-f347-4fea-8e90-19e808a787ab" (UID: "4466ec0b-f347-4fea-8e90-19e808a787ab"). InnerVolumeSpecName "kube-api-access-jprbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:04:19.721903 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:19.721805 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d31d10-7b9e-4668-8fcb-1a86406b079d-kube-api-access-d2n8x" (OuterVolumeSpecName: "kube-api-access-d2n8x") pod "94d31d10-7b9e-4668-8fcb-1a86406b079d" (UID: "94d31d10-7b9e-4668-8fcb-1a86406b079d"). InnerVolumeSpecName "kube-api-access-d2n8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:04:19.820700 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:19.820663 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4466ec0b-f347-4fea-8e90-19e808a787ab-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:19.820700 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:19.820695 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2n8x\" (UniqueName: \"kubernetes.io/projected/94d31d10-7b9e-4668-8fcb-1a86406b079d-kube-api-access-d2n8x\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:19.820700 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:19.820705 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jprbz\" (UniqueName: \"kubernetes.io/projected/4466ec0b-f347-4fea-8e90-19e808a787ab-kube-api-access-jprbz\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:20.039405 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.039372 2577 generic.go:356] "Generic (PLEG): container finished" podID="94d31d10-7b9e-4668-8fcb-1a86406b079d" containerID="1ec557a3ed0ca0240f46b93e981cbe54aeaaaeebe923ba9fcc0364cc5ff6e3ef" exitCode=0 Jul 31 15:04:20.039597 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.039417 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj" Jul 31 15:04:20.039597 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.039447 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj" event={"ID":"94d31d10-7b9e-4668-8fcb-1a86406b079d","Type":"ContainerDied","Data":"1ec557a3ed0ca0240f46b93e981cbe54aeaaaeebe923ba9fcc0364cc5ff6e3ef"} Jul 31 15:04:20.039597 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.039493 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj" event={"ID":"94d31d10-7b9e-4668-8fcb-1a86406b079d","Type":"ContainerDied","Data":"a5cbf0cf2e079eb4208132ef203a983571e85fa6ad834403bace094e06a1c087"} Jul 31 15:04:20.039597 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.039517 2577 scope.go:117] "RemoveContainer" containerID="1ec557a3ed0ca0240f46b93e981cbe54aeaaaeebe923ba9fcc0364cc5ff6e3ef" Jul 31 15:04:20.040729 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.040711 2577 generic.go:356] "Generic (PLEG): container finished" podID="4466ec0b-f347-4fea-8e90-19e808a787ab" containerID="df15352eb37fe0649896c77e39702c162430ccf3c7f001c1c83656722ea3e2a1" exitCode=0 Jul 31 15:04:20.040832 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.040817 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh" Jul 31 15:04:20.040930 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.040829 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh" event={"ID":"4466ec0b-f347-4fea-8e90-19e808a787ab","Type":"ContainerDied","Data":"df15352eb37fe0649896c77e39702c162430ccf3c7f001c1c83656722ea3e2a1"} Jul 31 15:04:20.040930 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.040893 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh" event={"ID":"4466ec0b-f347-4fea-8e90-19e808a787ab","Type":"ContainerDied","Data":"d9e2e312c1f2e25626f93cfaa5c62aa56d220a76518e359b0f8ed4b1eabce72a"} Jul 31 15:04:20.049758 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.049741 2577 scope.go:117] "RemoveContainer" containerID="1ec557a3ed0ca0240f46b93e981cbe54aeaaaeebe923ba9fcc0364cc5ff6e3ef" Jul 31 15:04:20.050100 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:04:20.050064 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ec557a3ed0ca0240f46b93e981cbe54aeaaaeebe923ba9fcc0364cc5ff6e3ef\": container with ID starting with 1ec557a3ed0ca0240f46b93e981cbe54aeaaaeebe923ba9fcc0364cc5ff6e3ef not found: ID does not exist" containerID="1ec557a3ed0ca0240f46b93e981cbe54aeaaaeebe923ba9fcc0364cc5ff6e3ef" Jul 31 15:04:20.050205 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.050100 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec557a3ed0ca0240f46b93e981cbe54aeaaaeebe923ba9fcc0364cc5ff6e3ef"} err="failed to get container status \"1ec557a3ed0ca0240f46b93e981cbe54aeaaaeebe923ba9fcc0364cc5ff6e3ef\": rpc error: code = NotFound desc = could not find container \"1ec557a3ed0ca0240f46b93e981cbe54aeaaaeebe923ba9fcc0364cc5ff6e3ef\": container with ID starting with 1ec557a3ed0ca0240f46b93e981cbe54aeaaaeebe923ba9fcc0364cc5ff6e3ef not found: ID does not exist" Jul 31 15:04:20.050205 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.050126 2577 scope.go:117] "RemoveContainer" containerID="df15352eb37fe0649896c77e39702c162430ccf3c7f001c1c83656722ea3e2a1" Jul 31 15:04:20.057388 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.057373 2577 scope.go:117] "RemoveContainer" containerID="df15352eb37fe0649896c77e39702c162430ccf3c7f001c1c83656722ea3e2a1" Jul 31 15:04:20.057631 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:04:20.057610 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df15352eb37fe0649896c77e39702c162430ccf3c7f001c1c83656722ea3e2a1\": container with ID starting with df15352eb37fe0649896c77e39702c162430ccf3c7f001c1c83656722ea3e2a1 not found: ID does not exist" containerID="df15352eb37fe0649896c77e39702c162430ccf3c7f001c1c83656722ea3e2a1" Jul 31 15:04:20.057692 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.057637 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df15352eb37fe0649896c77e39702c162430ccf3c7f001c1c83656722ea3e2a1"} err="failed to get container status \"df15352eb37fe0649896c77e39702c162430ccf3c7f001c1c83656722ea3e2a1\": rpc error: code = NotFound desc = could not find container \"df15352eb37fe0649896c77e39702c162430ccf3c7f001c1c83656722ea3e2a1\": container with ID starting with df15352eb37fe0649896c77e39702c162430ccf3c7f001c1c83656722ea3e2a1 not found: ID does not exist" Jul 31 15:04:20.061900 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.061878 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh"] Jul 31 15:04:20.066073 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.066054 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-700618-7d6f9c45fb-v6bqh"] Jul 31 15:04:20.074512 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.074493 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj"] Jul 31 15:04:20.077816 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.077797 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-700618-5d448c948d-v68sj"] Jul 31 15:04:20.430079 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.430051 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4466ec0b-f347-4fea-8e90-19e808a787ab" path="/var/lib/kubelet/pods/4466ec0b-f347-4fea-8e90-19e808a787ab/volumes" Jul 31 15:04:20.430437 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:20.430389 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94d31d10-7b9e-4668-8fcb-1a86406b079d" path="/var/lib/kubelet/pods/94d31d10-7b9e-4668-8fcb-1a86406b079d/volumes" Jul 31 15:04:21.675419 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:21.675385 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:04:25.880389 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:25.880356 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck"] Jul 31 15:04:25.880939 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:25.880559 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" podUID="66bfb387-27d3-485c-971f-3c27d346b63d" containerName="istio-proxy" containerID="cri-o://a06fb64c3d2a53de0bad896cc70a9a92692b6e1b8f706231de246f2e146cf185" gracePeriod=30 Jul 31 15:04:26.867453 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:26.867408 2577 patch_prober.go:28] interesting pod/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 15:04:26.867620 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:26.867479 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" podUID="66bfb387-27d3-485c-971f-3c27d346b63d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 15:04:31.099654 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.099623 2577 generic.go:356] "Generic (PLEG): container finished" podID="66bfb387-27d3-485c-971f-3c27d346b63d" containerID="a06fb64c3d2a53de0bad896cc70a9a92692b6e1b8f706231de246f2e146cf185" exitCode=0 Jul 31 15:04:31.100095 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.099679 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" event={"ID":"66bfb387-27d3-485c-971f-3c27d346b63d","Type":"ContainerDied","Data":"a06fb64c3d2a53de0bad896cc70a9a92692b6e1b8f706231de246f2e146cf185"} Jul 31 15:04:31.100095 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.099742 2577 scope.go:117] "RemoveContainer" containerID="ea50df13998cb3887289657aa0409df5e53ee2fef9979495555ee29e71966bd0" Jul 31 15:04:31.128291 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.128268 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:04:31.316773 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.316679 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bfb387-27d3-485c-971f-3c27d346b63d-istio-podinfo\") pod \"66bfb387-27d3-485c-971f-3c27d346b63d\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " Jul 31 15:04:31.316773 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.316728 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-workload-socket\") pod \"66bfb387-27d3-485c-971f-3c27d346b63d\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " Jul 31 15:04:31.316773 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.316759 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66bfb387-27d3-485c-971f-3c27d346b63d-istio-token\") pod \"66bfb387-27d3-485c-971f-3c27d346b63d\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " Jul 31 15:04:31.317088 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.316792 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bfb387-27d3-485c-971f-3c27d346b63d-istiod-ca-cert\") pod \"66bfb387-27d3-485c-971f-3c27d346b63d\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " Jul 31 15:04:31.317088 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.316813 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-istio-envoy\") pod \"66bfb387-27d3-485c-971f-3c27d346b63d\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " Jul 31 15:04:31.317088 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.316874 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-workload-certs\") pod \"66bfb387-27d3-485c-971f-3c27d346b63d\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " Jul 31 15:04:31.317088 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.316914 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-istio-data\") pod \"66bfb387-27d3-485c-971f-3c27d346b63d\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " Jul 31 15:04:31.317088 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.316937 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bxcb\" (UniqueName: \"kubernetes.io/projected/66bfb387-27d3-485c-971f-3c27d346b63d-kube-api-access-7bxcb\") pod \"66bfb387-27d3-485c-971f-3c27d346b63d\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " Jul 31 15:04:31.317088 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.316963 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-credential-socket\") pod \"66bfb387-27d3-485c-971f-3c27d346b63d\" (UID: \"66bfb387-27d3-485c-971f-3c27d346b63d\") " Jul 31 15:04:31.317088 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.317003 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "66bfb387-27d3-485c-971f-3c27d346b63d" (UID: "66bfb387-27d3-485c-971f-3c27d346b63d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:04:31.317442 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.317193 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "66bfb387-27d3-485c-971f-3c27d346b63d" (UID: "66bfb387-27d3-485c-971f-3c27d346b63d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:04:31.317442 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.317244 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-workload-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:31.317442 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.317297 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66bfb387-27d3-485c-971f-3c27d346b63d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "66bfb387-27d3-485c-971f-3c27d346b63d" (UID: "66bfb387-27d3-485c-971f-3c27d346b63d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:04:31.317442 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.317420 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "66bfb387-27d3-485c-971f-3c27d346b63d" (UID: "66bfb387-27d3-485c-971f-3c27d346b63d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:04:31.317718 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.317441 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-istio-data" (OuterVolumeSpecName: "istio-data") pod "66bfb387-27d3-485c-971f-3c27d346b63d" (UID: "66bfb387-27d3-485c-971f-3c27d346b63d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:04:31.319103 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.319069 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66bfb387-27d3-485c-971f-3c27d346b63d-istio-token" (OuterVolumeSpecName: "istio-token") pod "66bfb387-27d3-485c-971f-3c27d346b63d" (UID: "66bfb387-27d3-485c-971f-3c27d346b63d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:04:31.319210 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.319122 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "66bfb387-27d3-485c-971f-3c27d346b63d" (UID: "66bfb387-27d3-485c-971f-3c27d346b63d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:04:31.319210 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.319129 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66bfb387-27d3-485c-971f-3c27d346b63d-kube-api-access-7bxcb" (OuterVolumeSpecName: "kube-api-access-7bxcb") pod "66bfb387-27d3-485c-971f-3c27d346b63d" (UID: "66bfb387-27d3-485c-971f-3c27d346b63d"). InnerVolumeSpecName "kube-api-access-7bxcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:04:31.319210 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.319133 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66bfb387-27d3-485c-971f-3c27d346b63d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "66bfb387-27d3-485c-971f-3c27d346b63d" (UID: "66bfb387-27d3-485c-971f-3c27d346b63d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 15:04:31.417607 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.417579 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-workload-certs\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:31.417607 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.417600 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-istio-data\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:31.417607 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.417609 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bxcb\" (UniqueName: \"kubernetes.io/projected/66bfb387-27d3-485c-971f-3c27d346b63d-kube-api-access-7bxcb\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:31.417781 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.417618 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-credential-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:31.417781 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.417627 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bfb387-27d3-485c-971f-3c27d346b63d-istio-podinfo\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:31.417781 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.417635 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66bfb387-27d3-485c-971f-3c27d346b63d-istio-token\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:31.417781 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.417642 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bfb387-27d3-485c-971f-3c27d346b63d-istiod-ca-cert\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:31.417781 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:31.417649 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bfb387-27d3-485c-971f-3c27d346b63d-istio-envoy\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:32.105658 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:32.105626 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" event={"ID":"66bfb387-27d3-485c-971f-3c27d346b63d","Type":"ContainerDied","Data":"fb84f59d805154772173abf9fc03019473c72ad9dbf3aa0e0ed96389c84dc566"} Jul 31 15:04:32.105658 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:32.105661 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck" Jul 31 15:04:32.106160 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:32.105681 2577 scope.go:117] "RemoveContainer" containerID="a06fb64c3d2a53de0bad896cc70a9a92692b6e1b8f706231de246f2e146cf185" Jul 31 15:04:32.125539 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:32.125511 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck"] Jul 31 15:04:32.130739 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:32.130720 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-700618-openshift-default-7bcc45d99b-gmzck"] Jul 31 15:04:32.435126 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:32.435093 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66bfb387-27d3-485c-971f-3c27d346b63d" path="/var/lib/kubelet/pods/66bfb387-27d3-485c-971f-3c27d346b63d/volumes" Jul 31 15:04:41.704092 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:41.704057 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8"] Jul 31 15:04:41.704913 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:41.704879 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8" podUID="cfa4b272-5198-4716-b4ef-5c9240606f0e" containerName="payload-pre-processing" containerID="cri-o://eb43d8bf881920f0fbb7b51527377db882d3f45310ab41c785779545648c2a3b" gracePeriod=30 Jul 31 15:04:41.964361 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:41.964294 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8" Jul 31 15:04:42.010544 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:42.010514 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfa4b272-5198-4716-b4ef-5c9240606f0e-plugins-config-volume\") pod \"cfa4b272-5198-4716-b4ef-5c9240606f0e\" (UID: \"cfa4b272-5198-4716-b4ef-5c9240606f0e\") " Jul 31 15:04:42.010697 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:42.010614 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn6fl\" (UniqueName: \"kubernetes.io/projected/cfa4b272-5198-4716-b4ef-5c9240606f0e-kube-api-access-tn6fl\") pod \"cfa4b272-5198-4716-b4ef-5c9240606f0e\" (UID: \"cfa4b272-5198-4716-b4ef-5c9240606f0e\") " Jul 31 15:04:42.010914 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:42.010889 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa4b272-5198-4716-b4ef-5c9240606f0e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cfa4b272-5198-4716-b4ef-5c9240606f0e" (UID: "cfa4b272-5198-4716-b4ef-5c9240606f0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:04:42.012592 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:42.012573 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa4b272-5198-4716-b4ef-5c9240606f0e-kube-api-access-tn6fl" (OuterVolumeSpecName: "kube-api-access-tn6fl") pod "cfa4b272-5198-4716-b4ef-5c9240606f0e" (UID: "cfa4b272-5198-4716-b4ef-5c9240606f0e"). InnerVolumeSpecName "kube-api-access-tn6fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:04:42.111227 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:42.111191 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfa4b272-5198-4716-b4ef-5c9240606f0e-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:42.111227 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:42.111220 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn6fl\" (UniqueName: \"kubernetes.io/projected/cfa4b272-5198-4716-b4ef-5c9240606f0e-kube-api-access-tn6fl\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:42.149755 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:42.149725 2577 generic.go:356] "Generic (PLEG): container finished" podID="cfa4b272-5198-4716-b4ef-5c9240606f0e" containerID="eb43d8bf881920f0fbb7b51527377db882d3f45310ab41c785779545648c2a3b" exitCode=0 Jul 31 15:04:42.149900 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:42.149774 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8" Jul 31 15:04:42.149900 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:42.149776 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8" event={"ID":"cfa4b272-5198-4716-b4ef-5c9240606f0e","Type":"ContainerDied","Data":"eb43d8bf881920f0fbb7b51527377db882d3f45310ab41c785779545648c2a3b"} Jul 31 15:04:42.149991 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:42.149901 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8" event={"ID":"cfa4b272-5198-4716-b4ef-5c9240606f0e","Type":"ContainerDied","Data":"466bb52590ff4505a97c1c73298d5eb648333f3fd99eb00f3b036a15a3635423"} Jul 31 15:04:42.149991 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:42.149929 2577 scope.go:117] "RemoveContainer" containerID="eb43d8bf881920f0fbb7b51527377db882d3f45310ab41c785779545648c2a3b" Jul 31 15:04:42.158071 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:42.158056 2577 scope.go:117] "RemoveContainer" containerID="eb43d8bf881920f0fbb7b51527377db882d3f45310ab41c785779545648c2a3b" Jul 31 15:04:42.158319 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:04:42.158300 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb43d8bf881920f0fbb7b51527377db882d3f45310ab41c785779545648c2a3b\": container with ID starting with eb43d8bf881920f0fbb7b51527377db882d3f45310ab41c785779545648c2a3b not found: ID does not exist" containerID="eb43d8bf881920f0fbb7b51527377db882d3f45310ab41c785779545648c2a3b" Jul 31 15:04:42.158389 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:42.158330 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb43d8bf881920f0fbb7b51527377db882d3f45310ab41c785779545648c2a3b"} err="failed to get container status \"eb43d8bf881920f0fbb7b51527377db882d3f45310ab41c785779545648c2a3b\": rpc error: code = NotFound desc = could not find container \"eb43d8bf881920f0fbb7b51527377db882d3f45310ab41c785779545648c2a3b\": container with ID starting with eb43d8bf881920f0fbb7b51527377db882d3f45310ab41c785779545648c2a3b not found: ID does not exist" Jul 31 15:04:42.170119 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:42.170100 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8"] Jul 31 15:04:42.174299 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:42.174280 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2eeeef-5d4dd5b7cb-xkmm8"] Jul 31 15:04:42.430062 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:42.430030 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfa4b272-5198-4716-b4ef-5c9240606f0e" path="/var/lib/kubelet/pods/cfa4b272-5198-4716-b4ef-5c9240606f0e/volumes" Jul 31 15:04:43.873329 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:43.873292 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:04:48.139132 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:48.139101 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h"] Jul 31 15:04:48.139594 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:48.139404 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" podUID="d7dfc061-287f-40fd-82db-f50c8b698e12" containerName="istio-proxy" containerID="cri-o://64e2d3e675a669d612fb85f6195679aa2af4cd302a4787eaecc2cd5e413ee2fa" gracePeriod=30 Jul 31 15:04:48.999677 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:48.999647 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw"] Jul 31 15:04:48.999955 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:48.999942 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66bfb387-27d3-485c-971f-3c27d346b63d" containerName="istio-proxy" Jul 31 15:04:49.000006 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:48.999956 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="66bfb387-27d3-485c-971f-3c27d346b63d" containerName="istio-proxy" Jul 31 15:04:49.000006 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:48.999970 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4466ec0b-f347-4fea-8e90-19e808a787ab" containerName="payload-pre-processing" Jul 31 15:04:49.000006 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:48.999976 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4466ec0b-f347-4fea-8e90-19e808a787ab" containerName="payload-pre-processing" Jul 31 15:04:49.000006 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:48.999983 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa4b272-5198-4716-b4ef-5c9240606f0e" containerName="payload-pre-processing" Jul 31 15:04:49.000006 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:48.999988 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa4b272-5198-4716-b4ef-5c9240606f0e" containerName="payload-pre-processing" Jul 31 15:04:49.000006 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:48.999995 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66bfb387-27d3-485c-971f-3c27d346b63d" containerName="istio-proxy" Jul 31 15:04:49.000006 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.000000 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="66bfb387-27d3-485c-971f-3c27d346b63d" containerName="istio-proxy" Jul 31 15:04:49.000006 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.000005 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94d31d10-7b9e-4668-8fcb-1a86406b079d" containerName="payload-processing" Jul 31 15:04:49.000242 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.000010 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="94d31d10-7b9e-4668-8fcb-1a86406b079d" containerName="payload-processing" Jul 31 15:04:49.000242 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.000062 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="66bfb387-27d3-485c-971f-3c27d346b63d" containerName="istio-proxy" Jul 31 15:04:49.000242 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.000069 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="94d31d10-7b9e-4668-8fcb-1a86406b079d" containerName="payload-processing" Jul 31 15:04:49.000242 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.000075 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4466ec0b-f347-4fea-8e90-19e808a787ab" containerName="payload-pre-processing" Jul 31 15:04:49.000242 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.000081 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfa4b272-5198-4716-b4ef-5c9240606f0e" containerName="payload-pre-processing" Jul 31 15:04:49.004306 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.004287 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.006066 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.006049 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-9966b2-openshift-default-dockercfg-jdz9k\"" Jul 31 15:04:49.015055 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.015033 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw"] Jul 31 15:04:49.074696 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.074663 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-envoy\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.074696 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.074697 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-workload-socket\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.074948 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.074720 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znlzb\" (UniqueName: \"kubernetes.io/projected/8a78760f-ff69-4f1c-800f-8c9cac414b13-kube-api-access-znlzb\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.074948 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.074750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-token\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.074948 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.074817 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-data\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.074948 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.074867 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a78760f-ff69-4f1c-800f-8c9cac414b13-istiod-ca-cert\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.074948 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.074930 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-podinfo\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.075149 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.074970 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-workload-certs\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.075149 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.074996 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-credential-socket\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.175746 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.175708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-envoy\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.175746 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.175751 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-workload-socket\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.176295 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.175777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znlzb\" (UniqueName: \"kubernetes.io/projected/8a78760f-ff69-4f1c-800f-8c9cac414b13-kube-api-access-znlzb\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.176295 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.175818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-token\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.176295 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.175876 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-data\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.176295 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.175903 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a78760f-ff69-4f1c-800f-8c9cac414b13-istiod-ca-cert\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.176295 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.175952 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-podinfo\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.176295 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.176021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-workload-certs\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.176295 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.176062 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-credential-socket\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.176295 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.176261 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-workload-socket\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.176295 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.176279 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-data\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.176663 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.176330 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-workload-certs\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.176663 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.176421 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-credential-socket\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.176862 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.176820 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a78760f-ff69-4f1c-800f-8c9cac414b13-istiod-ca-cert\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.178291 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.178269 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-envoy\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.178402 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.178387 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-podinfo\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.184379 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.184359 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-token\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.184648 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.184630 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znlzb\" (UniqueName: \"kubernetes.io/projected/8a78760f-ff69-4f1c-800f-8c9cac414b13-kube-api-access-znlzb\") pod \"e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.313962 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.313836 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:49.448505 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.448476 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw"] Jul 31 15:04:49.451327 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:04:49.451300 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a78760f_ff69_4f1c_800f_8c9cac414b13.slice/crio-98540210cc015bd362c963f788535d20f6d7ab9b7e5a8524cd0d949b63eccdec WatchSource:0}: Error finding container 98540210cc015bd362c963f788535d20f6d7ab9b7e5a8524cd0d949b63eccdec: Status 404 returned error can't find the container with id 98540210cc015bd362c963f788535d20f6d7ab9b7e5a8524cd0d949b63eccdec Jul 31 15:04:49.453924 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.453892 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:04:49.454004 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.453959 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:04:49.454004 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:49.453995 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:04:50.190605 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:50.190566 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" event={"ID":"8a78760f-ff69-4f1c-800f-8c9cac414b13","Type":"ContainerStarted","Data":"0a4c6fc43b63230a2d770aa53e664e5f99aa4c7b17a9bec405243e53ebaa9a0a"} Jul 31 15:04:50.191070 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:50.190627 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" event={"ID":"8a78760f-ff69-4f1c-800f-8c9cac414b13","Type":"ContainerStarted","Data":"98540210cc015bd362c963f788535d20f6d7ab9b7e5a8524cd0d949b63eccdec"} Jul 31 15:04:50.211083 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:50.211034 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" podStartSLOduration=2.211018007 podStartE2EDuration="2.211018007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:04:50.210304656 +0000 UTC m=+3142.247127726" watchObservedRunningTime="2026-07-31 15:04:50.211018007 +0000 UTC m=+3142.247841074" Jul 31 15:04:50.314323 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:50.314228 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:50.319246 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:50.319217 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:51.195386 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:51.195357 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:51.196561 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:51.196540 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:04:53.385255 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.385225 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:04:53.515041 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.514948 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-envoy\") pod \"d7dfc061-287f-40fd-82db-f50c8b698e12\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " Jul 31 15:04:53.515041 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.514987 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-workload-certs\") pod \"d7dfc061-287f-40fd-82db-f50c8b698e12\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " Jul 31 15:04:53.515041 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.515022 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-credential-socket\") pod \"d7dfc061-287f-40fd-82db-f50c8b698e12\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " Jul 31 15:04:53.515041 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.515045 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-data\") pod \"d7dfc061-287f-40fd-82db-f50c8b698e12\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " Jul 31 15:04:53.515364 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.515066 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-podinfo\") pod \"d7dfc061-287f-40fd-82db-f50c8b698e12\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " Jul 31 15:04:53.515364 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.515087 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7dfc061-287f-40fd-82db-f50c8b698e12-istiod-ca-cert\") pod \"d7dfc061-287f-40fd-82db-f50c8b698e12\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " Jul 31 15:04:53.515364 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.515133 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-workload-socket\") pod \"d7dfc061-287f-40fd-82db-f50c8b698e12\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " Jul 31 15:04:53.515364 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.515167 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qnk4\" (UniqueName: \"kubernetes.io/projected/d7dfc061-287f-40fd-82db-f50c8b698e12-kube-api-access-9qnk4\") pod \"d7dfc061-287f-40fd-82db-f50c8b698e12\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " Jul 31 15:04:53.515364 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.515188 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-token\") pod \"d7dfc061-287f-40fd-82db-f50c8b698e12\" (UID: \"d7dfc061-287f-40fd-82db-f50c8b698e12\") " Jul 31 15:04:53.515364 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.515285 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d7dfc061-287f-40fd-82db-f50c8b698e12" (UID: "d7dfc061-287f-40fd-82db-f50c8b698e12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:04:53.515364 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.515316 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-data" (OuterVolumeSpecName: "istio-data") pod "d7dfc061-287f-40fd-82db-f50c8b698e12" (UID: "d7dfc061-287f-40fd-82db-f50c8b698e12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:04:53.515795 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.515461 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-data\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:53.515795 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.515480 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-workload-certs\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:53.515795 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.515474 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d7dfc061-287f-40fd-82db-f50c8b698e12" (UID: "d7dfc061-287f-40fd-82db-f50c8b698e12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:04:53.515795 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.515622 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d7dfc061-287f-40fd-82db-f50c8b698e12" (UID: "d7dfc061-287f-40fd-82db-f50c8b698e12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:04:53.515795 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.515713 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7dfc061-287f-40fd-82db-f50c8b698e12-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d7dfc061-287f-40fd-82db-f50c8b698e12" (UID: "d7dfc061-287f-40fd-82db-f50c8b698e12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:04:53.517340 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.517311 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d7dfc061-287f-40fd-82db-f50c8b698e12" (UID: "d7dfc061-287f-40fd-82db-f50c8b698e12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 15:04:53.517441 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.517361 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-token" (OuterVolumeSpecName: "istio-token") pod "d7dfc061-287f-40fd-82db-f50c8b698e12" (UID: "d7dfc061-287f-40fd-82db-f50c8b698e12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:04:53.517602 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.517581 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7dfc061-287f-40fd-82db-f50c8b698e12-kube-api-access-9qnk4" (OuterVolumeSpecName: "kube-api-access-9qnk4") pod "d7dfc061-287f-40fd-82db-f50c8b698e12" (UID: "d7dfc061-287f-40fd-82db-f50c8b698e12"). InnerVolumeSpecName "kube-api-access-9qnk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:04:53.517670 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.517581 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d7dfc061-287f-40fd-82db-f50c8b698e12" (UID: "d7dfc061-287f-40fd-82db-f50c8b698e12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:04:53.616240 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.616201 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-credential-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:53.616240 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.616234 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-podinfo\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:53.616240 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.616247 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7dfc061-287f-40fd-82db-f50c8b698e12-istiod-ca-cert\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:53.616479 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.616256 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-workload-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:53.616479 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.616265 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qnk4\" (UniqueName: \"kubernetes.io/projected/d7dfc061-287f-40fd-82db-f50c8b698e12-kube-api-access-9qnk4\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:53.616479 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.616275 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-token\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:53.616479 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.616283 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7dfc061-287f-40fd-82db-f50c8b698e12-istio-envoy\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:04:53.825302 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.825214 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd"] Jul 31 15:04:53.825553 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.825539 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7dfc061-287f-40fd-82db-f50c8b698e12" containerName="istio-proxy" Jul 31 15:04:53.825596 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.825555 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7dfc061-287f-40fd-82db-f50c8b698e12" containerName="istio-proxy" Jul 31 15:04:53.825631 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.825615 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="66bfb387-27d3-485c-971f-3c27d346b63d" containerName="istio-proxy" Jul 31 15:04:53.825631 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.825626 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7dfc061-287f-40fd-82db-f50c8b698e12" containerName="istio-proxy" Jul 31 15:04:53.825694 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.825634 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7dfc061-287f-40fd-82db-f50c8b698e12" containerName="istio-proxy" Jul 31 15:04:53.828957 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.828941 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd" Jul 31 15:04:53.830959 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.830939 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-9966b2-dockercfg-8g9rw\"" Jul 31 15:04:53.831186 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.831171 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-9966b2\"" Jul 31 15:04:53.836835 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.836814 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd"] Jul 31 15:04:53.865012 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.864978 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4"] Jul 31 15:04:53.865311 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.865297 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7dfc061-287f-40fd-82db-f50c8b698e12" containerName="istio-proxy" Jul 31 15:04:53.865377 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.865313 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7dfc061-287f-40fd-82db-f50c8b698e12" containerName="istio-proxy" Jul 31 15:04:53.868094 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.868079 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4" Jul 31 15:04:53.875747 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.875708 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4"] Jul 31 15:04:53.919308 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.919270 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69x27\" (UniqueName: \"kubernetes.io/projected/06e3de25-5249-4bfd-bf75-a0c0b013b660-kube-api-access-69x27\") pod \"payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd\" (UID: \"06e3de25-5249-4bfd-bf75-a0c0b013b660\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd" Jul 31 15:04:53.919512 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.919326 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92nvd\" (UniqueName: \"kubernetes.io/projected/89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f-kube-api-access-92nvd\") pod \"payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4\" (UID: \"89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4" Jul 31 15:04:53.919512 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.919436 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e3de25-5249-4bfd-bf75-a0c0b013b660-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd\" (UID: \"06e3de25-5249-4bfd-bf75-a0c0b013b660\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd" Jul 31 15:04:53.919512 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:53.919481 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4\" (UID: \"89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4" Jul 31 15:04:54.020151 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.020108 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69x27\" (UniqueName: \"kubernetes.io/projected/06e3de25-5249-4bfd-bf75-a0c0b013b660-kube-api-access-69x27\") pod \"payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd\" (UID: \"06e3de25-5249-4bfd-bf75-a0c0b013b660\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd" Jul 31 15:04:54.020346 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.020161 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92nvd\" (UniqueName: \"kubernetes.io/projected/89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f-kube-api-access-92nvd\") pod \"payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4\" (UID: \"89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4" Jul 31 15:04:54.020346 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.020211 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e3de25-5249-4bfd-bf75-a0c0b013b660-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd\" (UID: \"06e3de25-5249-4bfd-bf75-a0c0b013b660\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd" Jul 31 15:04:54.020346 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.020240 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4\" (UID: \"89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4" Jul 31 15:04:54.021040 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.021009 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e3de25-5249-4bfd-bf75-a0c0b013b660-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd\" (UID: \"06e3de25-5249-4bfd-bf75-a0c0b013b660\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd" Jul 31 15:04:54.021135 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.021101 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4\" (UID: \"89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4" Jul 31 15:04:54.031134 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.031110 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92nvd\" (UniqueName: \"kubernetes.io/projected/89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f-kube-api-access-92nvd\") pod \"payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4\" (UID: \"89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4" Jul 31 15:04:54.031219 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.031115 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69x27\" (UniqueName: \"kubernetes.io/projected/06e3de25-5249-4bfd-bf75-a0c0b013b660-kube-api-access-69x27\") pod \"payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd\" (UID: \"06e3de25-5249-4bfd-bf75-a0c0b013b660\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd" Jul 31 15:04:54.141116 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.141082 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd" Jul 31 15:04:54.178534 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.178500 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4" Jul 31 15:04:54.214550 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.214512 2577 generic.go:356] "Generic (PLEG): container finished" podID="d7dfc061-287f-40fd-82db-f50c8b698e12" containerID="64e2d3e675a669d612fb85f6195679aa2af4cd302a4787eaecc2cd5e413ee2fa" exitCode=0 Jul 31 15:04:54.214736 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.214602 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" event={"ID":"d7dfc061-287f-40fd-82db-f50c8b698e12","Type":"ContainerDied","Data":"64e2d3e675a669d612fb85f6195679aa2af4cd302a4787eaecc2cd5e413ee2fa"} Jul 31 15:04:54.214736 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.214641 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" event={"ID":"d7dfc061-287f-40fd-82db-f50c8b698e12","Type":"ContainerDied","Data":"15f5e9d417b382755f99292cb564ef6ceb4240c95fcf68ba22af6df0c575e872"} Jul 31 15:04:54.214736 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.214665 2577 scope.go:117] "RemoveContainer" containerID="64e2d3e675a669d612fb85f6195679aa2af4cd302a4787eaecc2cd5e413ee2fa" Jul 31 15:04:54.214931 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.214792 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h" Jul 31 15:04:54.226632 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.226608 2577 scope.go:117] "RemoveContainer" containerID="2aa5772b6bdda49d84a77ff3b270c67cf8fa18ff54f45b755816b49bd6f316ea" Jul 31 15:04:54.239001 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.238559 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h"] Jul 31 15:04:54.239001 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.238765 2577 scope.go:117] "RemoveContainer" containerID="64e2d3e675a669d612fb85f6195679aa2af4cd302a4787eaecc2cd5e413ee2fa" Jul 31 15:04:54.239331 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:04:54.239298 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64e2d3e675a669d612fb85f6195679aa2af4cd302a4787eaecc2cd5e413ee2fa\": container with ID starting with 64e2d3e675a669d612fb85f6195679aa2af4cd302a4787eaecc2cd5e413ee2fa not found: ID does not exist" containerID="64e2d3e675a669d612fb85f6195679aa2af4cd302a4787eaecc2cd5e413ee2fa" Jul 31 15:04:54.239419 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.239332 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64e2d3e675a669d612fb85f6195679aa2af4cd302a4787eaecc2cd5e413ee2fa"} err="failed to get container status \"64e2d3e675a669d612fb85f6195679aa2af4cd302a4787eaecc2cd5e413ee2fa\": rpc error: code = NotFound desc = could not find container \"64e2d3e675a669d612fb85f6195679aa2af4cd302a4787eaecc2cd5e413ee2fa\": container with ID starting with 64e2d3e675a669d612fb85f6195679aa2af4cd302a4787eaecc2cd5e413ee2fa not found: ID does not exist" Jul 31 15:04:54.239419 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.239358 2577 scope.go:117] "RemoveContainer" containerID="2aa5772b6bdda49d84a77ff3b270c67cf8fa18ff54f45b755816b49bd6f316ea" Jul 31 15:04:54.239632 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:04:54.239590 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aa5772b6bdda49d84a77ff3b270c67cf8fa18ff54f45b755816b49bd6f316ea\": container with ID starting with 2aa5772b6bdda49d84a77ff3b270c67cf8fa18ff54f45b755816b49bd6f316ea not found: ID does not exist" containerID="2aa5772b6bdda49d84a77ff3b270c67cf8fa18ff54f45b755816b49bd6f316ea" Jul 31 15:04:54.239632 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.239617 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa5772b6bdda49d84a77ff3b270c67cf8fa18ff54f45b755816b49bd6f316ea"} err="failed to get container status \"2aa5772b6bdda49d84a77ff3b270c67cf8fa18ff54f45b755816b49bd6f316ea\": rpc error: code = NotFound desc = could not find container \"2aa5772b6bdda49d84a77ff3b270c67cf8fa18ff54f45b755816b49bd6f316ea\": container with ID starting with 2aa5772b6bdda49d84a77ff3b270c67cf8fa18ff54f45b755816b49bd6f316ea not found: ID does not exist" Jul 31 15:04:54.259922 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.248556 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2eeeef-openshift-default-5fc59dbfbb-qlh8h"] Jul 31 15:04:54.307068 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.307037 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd"] Jul 31 15:04:54.310709 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:04:54.310669 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e3de25_5249_4bfd_bf75_a0c0b013b660.slice/crio-b0779eb10c4cbf216c5e402ee90a7a9bcd8ffcdd02c8dfb4bd4250a591738685 WatchSource:0}: Error finding container b0779eb10c4cbf216c5e402ee90a7a9bcd8ffcdd02c8dfb4bd4250a591738685: Status 404 returned error can't find the container with id b0779eb10c4cbf216c5e402ee90a7a9bcd8ffcdd02c8dfb4bd4250a591738685 Jul 31 15:04:54.360188 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.360099 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4"] Jul 31 15:04:54.362652 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:04:54.362629 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b4ba06_83bd_4a7f_b3a6_1d910d3ef42f.slice/crio-2a9c2b66101f0a2e50f3e847af51f5c07b894bfa8c760693191f23b9aeb0df9d WatchSource:0}: Error finding container 2a9c2b66101f0a2e50f3e847af51f5c07b894bfa8c760693191f23b9aeb0df9d: Status 404 returned error can't find the container with id 2a9c2b66101f0a2e50f3e847af51f5c07b894bfa8c760693191f23b9aeb0df9d Jul 31 15:04:54.432101 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:54.432058 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7dfc061-287f-40fd-82db-f50c8b698e12" path="/var/lib/kubelet/pods/d7dfc061-287f-40fd-82db-f50c8b698e12/volumes" Jul 31 15:04:55.221342 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:55.221305 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd" event={"ID":"06e3de25-5249-4bfd-bf75-a0c0b013b660","Type":"ContainerStarted","Data":"bf43374897623cc4a6416a22e68ffbbe94d86f3a46b47fe76c9cae42f5e84ac6"} Jul 31 15:04:55.221521 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:55.221367 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd" event={"ID":"06e3de25-5249-4bfd-bf75-a0c0b013b660","Type":"ContainerStarted","Data":"b0779eb10c4cbf216c5e402ee90a7a9bcd8ffcdd02c8dfb4bd4250a591738685"} Jul 31 15:04:55.221521 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:55.221404 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd" Jul 31 15:04:55.222794 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:55.222768 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4" event={"ID":"89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f","Type":"ContainerStarted","Data":"a376b967aaef9e9f34620be7dd7f80c29892ffe55b238bedac0256534195d325"} Jul 31 15:04:55.222908 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:55.222816 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4" event={"ID":"89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f","Type":"ContainerStarted","Data":"2a9c2b66101f0a2e50f3e847af51f5c07b894bfa8c760693191f23b9aeb0df9d"} Jul 31 15:04:55.222908 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:55.222864 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4" Jul 31 15:04:55.239431 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:55.239386 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd" podStartSLOduration=2.239371813 podStartE2EDuration="2.239371813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:04:55.237966046 +0000 UTC m=+3147.274789108" watchObservedRunningTime="2026-07-31 15:04:55.239371813 +0000 UTC m=+3147.276194880" Jul 31 15:04:55.253885 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:55.253817 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4" podStartSLOduration=2.25379835 podStartE2EDuration="2.25379835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:04:55.251709615 +0000 UTC m=+3147.288532682" watchObservedRunningTime="2026-07-31 15:04:55.25379835 +0000 UTC m=+3147.290621422" Jul 31 15:04:55.674619 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:04:55.674580 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:05:05.276766 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:05.276730 2577 generic.go:356] "Generic (PLEG): container finished" podID="8a78760f-ff69-4f1c-800f-8c9cac414b13" containerID="0a4c6fc43b63230a2d770aa53e664e5f99aa4c7b17a9bec405243e53ebaa9a0a" exitCode=0 Jul 31 15:05:05.277329 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:05.276809 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" event={"ID":"8a78760f-ff69-4f1c-800f-8c9cac414b13","Type":"ContainerDied","Data":"0a4c6fc43b63230a2d770aa53e664e5f99aa4c7b17a9bec405243e53ebaa9a0a"} Jul 31 15:05:05.277413 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:05.277352 2577 scope.go:117] "RemoveContainer" containerID="0a4c6fc43b63230a2d770aa53e664e5f99aa4c7b17a9bec405243e53ebaa9a0a" Jul 31 15:05:05.278442 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:05.278414 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:05:05.278519 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:05.278496 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:05:05.278575 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:05.278561 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:05:06.196300 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:06.196262 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:05:06.232763 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:06.231787 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4" Jul 31 15:05:06.233597 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:06.233070 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd" Jul 31 15:05:06.284290 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:06.284251 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" event={"ID":"8a78760f-ff69-4f1c-800f-8c9cac414b13","Type":"ContainerStarted","Data":"d92bf2f2e03a81889e37d5367d4e800bdadf064d3bbde13e02b40b0ce41d1ca2"} Jul 31 15:05:06.314402 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:06.314374 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:05:06.315327 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:06.315299 2577 patch_prober.go:28] interesting pod/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 31 15:05:06.315456 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:06.315371 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" podUID="8a78760f-ff69-4f1c-800f-8c9cac414b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 31 15:05:07.315240 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:07.315209 2577 patch_prober.go:28] interesting pod/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 31 15:05:07.315622 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:07.315258 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" podUID="8a78760f-ff69-4f1c-800f-8c9cac414b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 31 15:05:08.314836 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:08.314802 2577 patch_prober.go:28] interesting pod/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 31 15:05:08.315024 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:08.314869 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" podUID="8a78760f-ff69-4f1c-800f-8c9cac414b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 31 15:05:09.314664 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:09.314630 2577 patch_prober.go:28] interesting pod/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 31 15:05:09.315110 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:09.314683 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" podUID="8a78760f-ff69-4f1c-800f-8c9cac414b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 31 15:05:10.314327 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:10.314286 2577 patch_prober.go:28] interesting pod/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 31 15:05:10.314505 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:10.314378 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" podUID="8a78760f-ff69-4f1c-800f-8c9cac414b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 31 15:05:11.314468 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:11.314433 2577 patch_prober.go:28] interesting pod/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 31 15:05:11.314913 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:11.314485 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" podUID="8a78760f-ff69-4f1c-800f-8c9cac414b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 31 15:05:12.318494 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:12.318463 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:05:12.318901 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:12.318726 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:05:12.319445 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:12.319429 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:05:17.864165 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:17.864129 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv"] Jul 31 15:05:17.867007 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:17.866986 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:17.869073 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:17.869043 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-79a3ee-openshift-default-dockercfg-pp4d5\"" Jul 31 15:05:17.878145 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:17.878123 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv"] Jul 31 15:05:17.952453 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:17.952420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-podinfo\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:17.952704 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:17.952680 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-credential-socket\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:17.952807 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:17.952762 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81b6908-3ef9-407b-8f6b-28b7bc428311-istiod-ca-cert\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:17.952898 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:17.952835 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qd4w\" (UniqueName: \"kubernetes.io/projected/d81b6908-3ef9-407b-8f6b-28b7bc428311-kube-api-access-8qd4w\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:17.952898 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:17.952887 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-workload-socket\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:17.953016 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:17.952917 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-envoy\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:17.953016 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:17.952970 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-workload-certs\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:17.953016 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:17.952996 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-data\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:17.953237 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:17.953210 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-token\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.054026 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.053991 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-token\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.054227 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.054042 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-podinfo\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.054227 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.054079 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-credential-socket\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.054227 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.054116 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81b6908-3ef9-407b-8f6b-28b7bc428311-istiod-ca-cert\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.054227 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.054147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qd4w\" (UniqueName: \"kubernetes.io/projected/d81b6908-3ef9-407b-8f6b-28b7bc428311-kube-api-access-8qd4w\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.054432 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.054249 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-workload-socket\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.054432 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.054288 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-envoy\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.054432 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.054418 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-workload-certs\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.054567 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.054456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-data\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.054732 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.054706 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-workload-socket\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.054898 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.054750 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-workload-certs\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.054898 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.054820 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-credential-socket\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.054898 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.054868 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-data\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.055083 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.055005 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81b6908-3ef9-407b-8f6b-28b7bc428311-istiod-ca-cert\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.056581 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.056562 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-envoy\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.056701 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.056686 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-podinfo\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.060650 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.060628 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-token\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.060911 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.060893 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qd4w\" (UniqueName: \"kubernetes.io/projected/d81b6908-3ef9-407b-8f6b-28b7bc428311-kube-api-access-8qd4w\") pod \"e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.177133 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.177107 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:18.310952 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.310919 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv"] Jul 31 15:05:18.316643 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:05:18.316612 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81b6908_3ef9_407b_8f6b_28b7bc428311.slice/crio-3f33a61a3cd95087f3929acbd9929bbd883a64709564b740d4f5409970bbe9a7 WatchSource:0}: Error finding container 3f33a61a3cd95087f3929acbd9929bbd883a64709564b740d4f5409970bbe9a7: Status 404 returned error can't find the container with id 3f33a61a3cd95087f3929acbd9929bbd883a64709564b740d4f5409970bbe9a7 Jul 31 15:05:18.319103 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.319064 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:05:18.319177 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.319153 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:05:18.319217 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.319193 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:05:18.342071 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:18.342048 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" event={"ID":"d81b6908-3ef9-407b-8f6b-28b7bc428311","Type":"ContainerStarted","Data":"3f33a61a3cd95087f3929acbd9929bbd883a64709564b740d4f5409970bbe9a7"} Jul 31 15:05:19.347419 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:19.347383 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" event={"ID":"d81b6908-3ef9-407b-8f6b-28b7bc428311","Type":"ContainerStarted","Data":"052e11ed3c4ba642cb17826ad2502ac23bea8256628c4607e3ac529c86f9bf67"} Jul 31 15:05:19.366412 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:19.366362 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" podStartSLOduration=2.366348419 podStartE2EDuration="2.366348419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:05:19.363593247 +0000 UTC m=+3171.400416317" watchObservedRunningTime="2026-07-31 15:05:19.366348419 +0000 UTC m=+3171.403171486" Jul 31 15:05:20.178204 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:20.178161 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:20.183593 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:20.183566 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:20.352519 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:20.352485 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:20.353460 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:20.353436 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:25.069059 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:25.069023 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:05:35.354108 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:35.354000 2577 patch_prober.go:28] interesting pod/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 31 15:05:35.354108 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:35.354060 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" podUID="d81b6908-3ef9-407b-8f6b-28b7bc428311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 31 15:05:35.423980 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:35.423946 2577 generic.go:356] "Generic (PLEG): container finished" podID="d81b6908-3ef9-407b-8f6b-28b7bc428311" containerID="052e11ed3c4ba642cb17826ad2502ac23bea8256628c4607e3ac529c86f9bf67" exitCode=0 Jul 31 15:05:35.424140 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:35.423994 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" event={"ID":"d81b6908-3ef9-407b-8f6b-28b7bc428311","Type":"ContainerDied","Data":"052e11ed3c4ba642cb17826ad2502ac23bea8256628c4607e3ac529c86f9bf67"} Jul 31 15:05:35.424388 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:35.424375 2577 scope.go:117] "RemoveContainer" containerID="052e11ed3c4ba642cb17826ad2502ac23bea8256628c4607e3ac529c86f9bf67" Jul 31 15:05:35.425388 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:35.425354 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:05:35.425482 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:35.425445 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:05:35.425531 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:35.425480 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:05:36.434333 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:36.434290 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" event={"ID":"d81b6908-3ef9-407b-8f6b-28b7bc428311","Type":"ContainerStarted","Data":"bdb72d107987cb93db5ac7b4923d965a3c73d8fb5f049f180786d77357184f77"} Jul 31 15:05:37.177586 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:37.177546 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:37.177883 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:37.177833 2577 patch_prober.go:28] interesting pod/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 31 15:05:37.177936 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:37.177921 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" podUID="d81b6908-3ef9-407b-8f6b-28b7bc428311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 31 15:05:38.177541 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:38.177497 2577 patch_prober.go:28] interesting pod/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 31 15:05:38.177979 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:38.177570 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" podUID="d81b6908-3ef9-407b-8f6b-28b7bc428311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 31 15:05:39.177740 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:39.177700 2577 patch_prober.go:28] interesting pod/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 31 15:05:39.178200 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:39.177759 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" podUID="d81b6908-3ef9-407b-8f6b-28b7bc428311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 31 15:05:40.177561 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:40.177522 2577 patch_prober.go:28] interesting pod/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 31 15:05:40.177741 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:40.177579 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" podUID="d81b6908-3ef9-407b-8f6b-28b7bc428311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 31 15:05:41.178215 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:41.178177 2577 patch_prober.go:28] interesting pod/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 31 15:05:41.178644 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:41.178247 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" podUID="d81b6908-3ef9-407b-8f6b-28b7bc428311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 31 15:05:42.181522 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:42.181492 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:42.181939 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:42.181752 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:42.182466 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:42.182448 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:05:51.978642 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:05:51.978608 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:06:36.079663 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:06:36.079570 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:07:02.082171 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:07:02.082134 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:07:27.984218 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:07:27.984187 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:07:39.580740 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:07:39.580695 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:07:54.771957 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:07:54.771879 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:08:06.479635 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:08:06.479605 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:08:21.575925 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:08:21.575891 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:08:32.283030 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:08:32.282994 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:08:47.870791 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:08:47.870755 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:08:59.771574 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:08:59.771528 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:09:14.370051 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:09:14.370013 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:09:26.268116 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:09:26.268040 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:09:40.886571 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:09:40.886539 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:09:51.075340 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:09:51.075301 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:10:05.584514 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:10:05.584475 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:10:27.475436 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:10:27.475404 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:11:03.577686 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:11:03.577651 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:11:14.191409 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:11:14.191372 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:11:32.674816 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:11:32.674769 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:11:43.076685 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:11:43.076648 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:11:54.988304 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:11:54.988261 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:12:25.680469 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:12:25.680385 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:13:06.184251 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:06.184217 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:13:14.682145 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:14.682111 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:13:30.777485 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:30.777449 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:13:39.870230 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:39.870193 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:13:57.652101 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.652064 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v"] Jul 31 15:13:57.654894 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.654875 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.656828 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.656809 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-58af04-openshift-default-dockercfg-m7q6m\"" Jul 31 15:13:57.665491 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.665468 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v"] Jul 31 15:13:57.726888 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.726859 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-token\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.727015 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.726892 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de459d3d-e0b4-4810-a31b-666efc74cae2-istiod-ca-cert\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.727015 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.726918 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-envoy\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.727015 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.726935 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-data\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.727015 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.726951 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfblb\" (UniqueName: \"kubernetes.io/projected/de459d3d-e0b4-4810-a31b-666efc74cae2-kube-api-access-wfblb\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.727167 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.727011 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-workload-certs\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.727167 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.727062 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-credential-socket\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.727167 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.727102 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-workload-socket\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.727167 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.727126 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-podinfo\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.827711 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.827678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-token\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.827711 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.827717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de459d3d-e0b4-4810-a31b-666efc74cae2-istiod-ca-cert\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.828000 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.827744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-envoy\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.828000 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.827769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-data\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.828000 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.827794 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfblb\" (UniqueName: \"kubernetes.io/projected/de459d3d-e0b4-4810-a31b-666efc74cae2-kube-api-access-wfblb\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.828000 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.827835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-workload-certs\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.828000 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.827907 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-credential-socket\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.828255 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.828132 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-workload-socket\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.828255 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.828171 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-podinfo\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.828363 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.828301 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-data\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.828422 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.828353 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-workload-certs\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.828422 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.828398 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-credential-socket\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.828507 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.828445 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-workload-socket\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.828581 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.828563 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de459d3d-e0b4-4810-a31b-666efc74cae2-istiod-ca-cert\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.830180 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.830157 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-envoy\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.830447 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.830430 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-podinfo\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.834855 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.834818 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-token\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.834958 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.834928 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfblb\" (UniqueName: \"kubernetes.io/projected/de459d3d-e0b4-4810-a31b-666efc74cae2-kube-api-access-wfblb\") pod \"e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:57.965747 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:57.965658 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:58.098126 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:58.098098 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v"] Jul 31 15:13:58.100645 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:13:58.100618 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde459d3d_e0b4_4810_a31b_666efc74cae2.slice/crio-35f3db4f4a57d9738f33116dc27d180143428b6e63d6cf0a52a3a4c3bdd71b55 WatchSource:0}: Error finding container 35f3db4f4a57d9738f33116dc27d180143428b6e63d6cf0a52a3a4c3bdd71b55: Status 404 returned error can't find the container with id 35f3db4f4a57d9738f33116dc27d180143428b6e63d6cf0a52a3a4c3bdd71b55 Jul 31 15:13:58.102780 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:58.102750 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:13:58.102893 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:58.102809 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:13:58.102893 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:58.102864 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:13:58.660509 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:58.660470 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" event={"ID":"de459d3d-e0b4-4810-a31b-666efc74cae2","Type":"ContainerStarted","Data":"dfe7e16bad0561d3b01f6e62d664f5e6215ca011fb31f7199e094b0fe07eaae5"} Jul 31 15:13:58.660968 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:58.660527 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" event={"ID":"de459d3d-e0b4-4810-a31b-666efc74cae2","Type":"ContainerStarted","Data":"35f3db4f4a57d9738f33116dc27d180143428b6e63d6cf0a52a3a4c3bdd71b55"} Jul 31 15:13:58.679497 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:58.679451 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" podStartSLOduration=1.679436061 podStartE2EDuration="1.679436061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:13:58.677581015 +0000 UTC m=+3690.714404082" watchObservedRunningTime="2026-07-31 15:13:58.679436061 +0000 UTC m=+3690.716259128" Jul 31 15:13:58.966672 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:58.966592 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:13:59.971350 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:13:59.971318 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:14:00.669824 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:00.669789 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:14:00.670714 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:00.670698 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:14:02.448966 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:02.448934 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh"] Jul 31 15:14:02.451279 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:02.451264 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh" Jul 31 15:14:02.454360 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:02.454327 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-58af04\"" Jul 31 15:14:02.454486 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:02.454424 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-58af04-dockercfg-2h76c\"" Jul 31 15:14:02.463759 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:02.463733 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh"] Jul 31 15:14:02.570168 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:02.570131 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94wwg\" (UniqueName: \"kubernetes.io/projected/4feaae1a-54a5-4fc7-843e-d0c82255c13a-kube-api-access-94wwg\") pod \"payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh\" (UID: \"4feaae1a-54a5-4fc7-843e-d0c82255c13a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh" Jul 31 15:14:02.570340 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:02.570258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4feaae1a-54a5-4fc7-843e-d0c82255c13a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh\" (UID: \"4feaae1a-54a5-4fc7-843e-d0c82255c13a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh" Jul 31 15:14:02.671321 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:02.671289 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94wwg\" (UniqueName: \"kubernetes.io/projected/4feaae1a-54a5-4fc7-843e-d0c82255c13a-kube-api-access-94wwg\") pod \"payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh\" (UID: \"4feaae1a-54a5-4fc7-843e-d0c82255c13a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh" Jul 31 15:14:02.671463 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:02.671343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4feaae1a-54a5-4fc7-843e-d0c82255c13a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh\" (UID: \"4feaae1a-54a5-4fc7-843e-d0c82255c13a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh" Jul 31 15:14:02.672061 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:02.672043 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4feaae1a-54a5-4fc7-843e-d0c82255c13a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh\" (UID: \"4feaae1a-54a5-4fc7-843e-d0c82255c13a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh" Jul 31 15:14:02.681108 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:02.681087 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94wwg\" (UniqueName: \"kubernetes.io/projected/4feaae1a-54a5-4fc7-843e-d0c82255c13a-kube-api-access-94wwg\") pod \"payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh\" (UID: \"4feaae1a-54a5-4fc7-843e-d0c82255c13a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh" Jul 31 15:14:02.761573 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:02.761508 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh" Jul 31 15:14:02.886868 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:02.886822 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh"] Jul 31 15:14:02.889633 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:14:02.889599 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4feaae1a_54a5_4fc7_843e_d0c82255c13a.slice/crio-bd86feb37fc19dbad7c53a177fd590deac6a719e750c31341e4f67c90dfc8d96 WatchSource:0}: Error finding container bd86feb37fc19dbad7c53a177fd590deac6a719e750c31341e4f67c90dfc8d96: Status 404 returned error can't find the container with id bd86feb37fc19dbad7c53a177fd590deac6a719e750c31341e4f67c90dfc8d96 Jul 31 15:14:03.685407 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:03.685368 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh" event={"ID":"4feaae1a-54a5-4fc7-843e-d0c82255c13a","Type":"ContainerStarted","Data":"8e55bbf5ac148313dfe0094364ff2dce25530dee262e7587d8eb6a240dffae76"} Jul 31 15:14:03.685776 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:03.685428 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh" event={"ID":"4feaae1a-54a5-4fc7-843e-d0c82255c13a","Type":"ContainerStarted","Data":"bd86feb37fc19dbad7c53a177fd590deac6a719e750c31341e4f67c90dfc8d96"} Jul 31 15:14:03.685776 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:03.685506 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh" Jul 31 15:14:03.699752 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:03.699704 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh" podStartSLOduration=1.699688906 podStartE2EDuration="1.699688906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:14:03.698717985 +0000 UTC m=+3695.735541053" watchObservedRunningTime="2026-07-31 15:14:03.699688906 +0000 UTC m=+3695.736511972" Jul 31 15:14:04.869309 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:04.869273 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:14:14.692565 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:14.692528 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh" Jul 31 15:14:14.738171 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:14.738139 2577 generic.go:356] "Generic (PLEG): container finished" podID="de459d3d-e0b4-4810-a31b-666efc74cae2" containerID="dfe7e16bad0561d3b01f6e62d664f5e6215ca011fb31f7199e094b0fe07eaae5" exitCode=0 Jul 31 15:14:14.738329 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:14.738210 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" event={"ID":"de459d3d-e0b4-4810-a31b-666efc74cae2","Type":"ContainerDied","Data":"dfe7e16bad0561d3b01f6e62d664f5e6215ca011fb31f7199e094b0fe07eaae5"} Jul 31 15:14:14.738611 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:14.738596 2577 scope.go:117] "RemoveContainer" containerID="dfe7e16bad0561d3b01f6e62d664f5e6215ca011fb31f7199e094b0fe07eaae5" Jul 31 15:14:14.739638 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:14.739603 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:14:14.739723 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:14.739678 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:14:14.739723 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:14.739709 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 15:14:15.670630 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:15.670593 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:14:15.749053 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:15.749010 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" event={"ID":"de459d3d-e0b4-4810-a31b-666efc74cae2","Type":"ContainerStarted","Data":"d9f687152004c4fad4c1081426d2bf593f92324d3e161456928a0cebec88f382"} Jul 31 15:14:15.966772 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:15.966677 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:14:15.968036 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:15.968009 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 31 15:14:15.968151 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:15.968060 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" podUID="de459d3d-e0b4-4810-a31b-666efc74cae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 31 15:14:16.966344 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:16.966301 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 31 15:14:16.966761 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:16.966361 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" podUID="de459d3d-e0b4-4810-a31b-666efc74cae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 31 15:14:17.966648 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:17.966609 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 31 15:14:17.967091 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:17.966679 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" podUID="de459d3d-e0b4-4810-a31b-666efc74cae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 31 15:14:18.967008 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:18.966965 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 31 15:14:18.967402 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:18.967038 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" podUID="de459d3d-e0b4-4810-a31b-666efc74cae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 31 15:14:19.966931 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:19.966893 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 31 15:14:19.967119 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:19.966961 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" podUID="de459d3d-e0b4-4810-a31b-666efc74cae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 31 15:14:20.970920 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:20.970889 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:14:20.971399 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:20.971269 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:14:20.972693 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:20.972671 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:14:31.604018 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.603976 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd"] Jul 31 15:14:31.606747 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.606726 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd" Jul 31 15:14:31.609259 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.609232 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1c9d21\"" Jul 31 15:14:31.609388 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.609280 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1c9d21-dockercfg-qc8j9\"" Jul 31 15:14:31.616676 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.616655 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd"] Jul 31 15:14:31.626380 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.626359 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs"] Jul 31 15:14:31.628646 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.628622 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs" Jul 31 15:14:31.643416 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.643395 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs"] Jul 31 15:14:31.654645 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.654621 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4a38ac-d1ec-439e-aa0d-c498508794ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs\" (UID: \"3b4a38ac-d1ec-439e-aa0d-c498508794ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs" Jul 31 15:14:31.654740 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.654657 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxl44\" (UniqueName: \"kubernetes.io/projected/10dd78e5-a052-45f1-805c-8f7a9a1db0fa-kube-api-access-cxl44\") pod \"payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd\" (UID: \"10dd78e5-a052-45f1-805c-8f7a9a1db0fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd" Jul 31 15:14:31.654740 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.654721 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10dd78e5-a052-45f1-805c-8f7a9a1db0fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd\" (UID: \"10dd78e5-a052-45f1-805c-8f7a9a1db0fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd" Jul 31 15:14:31.654825 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.654788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwt9x\" (UniqueName: \"kubernetes.io/projected/3b4a38ac-d1ec-439e-aa0d-c498508794ee-kube-api-access-zwt9x\") pod \"payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs\" (UID: \"3b4a38ac-d1ec-439e-aa0d-c498508794ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs" Jul 31 15:14:31.755755 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.755717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4a38ac-d1ec-439e-aa0d-c498508794ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs\" (UID: \"3b4a38ac-d1ec-439e-aa0d-c498508794ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs" Jul 31 15:14:31.756042 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.756021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxl44\" (UniqueName: \"kubernetes.io/projected/10dd78e5-a052-45f1-805c-8f7a9a1db0fa-kube-api-access-cxl44\") pod \"payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd\" (UID: \"10dd78e5-a052-45f1-805c-8f7a9a1db0fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd" Jul 31 15:14:31.757192 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.757164 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10dd78e5-a052-45f1-805c-8f7a9a1db0fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd\" (UID: \"10dd78e5-a052-45f1-805c-8f7a9a1db0fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd" Jul 31 15:14:31.758265 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.758243 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4a38ac-d1ec-439e-aa0d-c498508794ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs\" (UID: \"3b4a38ac-d1ec-439e-aa0d-c498508794ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs" Jul 31 15:14:31.758746 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.758726 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10dd78e5-a052-45f1-805c-8f7a9a1db0fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd\" (UID: \"10dd78e5-a052-45f1-805c-8f7a9a1db0fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd" Jul 31 15:14:31.759001 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.758982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwt9x\" (UniqueName: \"kubernetes.io/projected/3b4a38ac-d1ec-439e-aa0d-c498508794ee-kube-api-access-zwt9x\") pod \"payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs\" (UID: \"3b4a38ac-d1ec-439e-aa0d-c498508794ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs" Jul 31 15:14:31.766735 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.766713 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwt9x\" (UniqueName: \"kubernetes.io/projected/3b4a38ac-d1ec-439e-aa0d-c498508794ee-kube-api-access-zwt9x\") pod \"payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs\" (UID: \"3b4a38ac-d1ec-439e-aa0d-c498508794ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs" Jul 31 15:14:31.766815 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.766731 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxl44\" (UniqueName: \"kubernetes.io/projected/10dd78e5-a052-45f1-805c-8f7a9a1db0fa-kube-api-access-cxl44\") pod \"payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd\" (UID: \"10dd78e5-a052-45f1-805c-8f7a9a1db0fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd" Jul 31 15:14:31.917912 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.917878 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd" Jul 31 15:14:31.938617 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:31.938594 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs" Jul 31 15:14:32.064395 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:32.064362 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd"] Jul 31 15:14:32.066792 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:14:32.066757 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10dd78e5_a052_45f1_805c_8f7a9a1db0fa.slice/crio-d25230325e25d7af9b034e7fea42b521718ffa10ba465cf55c5b340ff842974a WatchSource:0}: Error finding container d25230325e25d7af9b034e7fea42b521718ffa10ba465cf55c5b340ff842974a: Status 404 returned error can't find the container with id d25230325e25d7af9b034e7fea42b521718ffa10ba465cf55c5b340ff842974a Jul 31 15:14:32.086711 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:32.086689 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs"] Jul 31 15:14:32.089193 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:14:32.089158 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4a38ac_d1ec_439e_aa0d_c498508794ee.slice/crio-a3b8d1ba90aee2fa1bab37dd46c079e1dd6611a8a39a2a595682647900879b62 WatchSource:0}: Error finding container a3b8d1ba90aee2fa1bab37dd46c079e1dd6611a8a39a2a595682647900879b62: Status 404 returned error can't find the container with id a3b8d1ba90aee2fa1bab37dd46c079e1dd6611a8a39a2a595682647900879b62 Jul 31 15:14:32.838557 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:32.838513 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd" event={"ID":"10dd78e5-a052-45f1-805c-8f7a9a1db0fa","Type":"ContainerStarted","Data":"73dff7a42bc376eda4c7650ff8461fcf58ba71bef201d770a9798a20ef42ae10"} Jul 31 15:14:32.839061 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:32.838581 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd" event={"ID":"10dd78e5-a052-45f1-805c-8f7a9a1db0fa","Type":"ContainerStarted","Data":"d25230325e25d7af9b034e7fea42b521718ffa10ba465cf55c5b340ff842974a"} Jul 31 15:14:32.839061 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:32.838611 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd" Jul 31 15:14:32.840173 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:32.840146 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs" event={"ID":"3b4a38ac-d1ec-439e-aa0d-c498508794ee","Type":"ContainerStarted","Data":"00cd8611375c4752ba6d75895e53be2e42350433b7611da85ca790bd1ea47432"} Jul 31 15:14:32.840261 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:32.840198 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs" event={"ID":"3b4a38ac-d1ec-439e-aa0d-c498508794ee","Type":"ContainerStarted","Data":"a3b8d1ba90aee2fa1bab37dd46c079e1dd6611a8a39a2a595682647900879b62"} Jul 31 15:14:32.840261 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:32.840238 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs" Jul 31 15:14:32.852582 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:32.852396 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd" podStartSLOduration=1.852382162 podStartE2EDuration="1.852382162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:14:32.852341758 +0000 UTC m=+3724.889164819" watchObservedRunningTime="2026-07-31 15:14:32.852382162 +0000 UTC m=+3724.889205226" Jul 31 15:14:32.867283 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:32.867240 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs" podStartSLOduration=1.867226489 podStartE2EDuration="1.867226489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:14:32.86568043 +0000 UTC m=+3724.902503500" watchObservedRunningTime="2026-07-31 15:14:32.867226489 +0000 UTC m=+3724.904049556" Jul 31 15:14:34.490037 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:34.490002 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:14:43.847957 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:43.847927 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd" Jul 31 15:14:43.848482 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:14:43.848081 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs" Jul 31 15:15:01.839856 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:01.839806 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs_3b4a38ac-d1ec-439e-aa0d-c498508794ee/payload-processing/0.log" Jul 31 15:15:06.273670 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:06.273634 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:15:30.388461 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:30.388363 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:15:41.742115 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:41.742085 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs_3b4a38ac-d1ec-439e-aa0d-c498508794ee/payload-processing/0.log" Jul 31 15:15:46.994911 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:46.994868 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7"] Jul 31 15:15:46.998350 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:46.998327 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7" Jul 31 15:15:47.000393 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:47.000363 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-39573e-dockercfg-khwwj\"" Jul 31 15:15:47.000393 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:47.000371 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-39573e\"" Jul 31 15:15:47.006337 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:47.006316 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7"] Jul 31 15:15:47.148163 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:47.148127 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6cvh\" (UniqueName: \"kubernetes.io/projected/99add764-c1dd-428a-bb04-a6f4ed6599cf-kube-api-access-j6cvh\") pod \"payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7\" (UID: \"99add764-c1dd-428a-bb04-a6f4ed6599cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7" Jul 31 15:15:47.148354 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:47.148274 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99add764-c1dd-428a-bb04-a6f4ed6599cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7\" (UID: \"99add764-c1dd-428a-bb04-a6f4ed6599cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7" Jul 31 15:15:47.249739 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:47.249646 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6cvh\" (UniqueName: \"kubernetes.io/projected/99add764-c1dd-428a-bb04-a6f4ed6599cf-kube-api-access-j6cvh\") pod \"payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7\" (UID: \"99add764-c1dd-428a-bb04-a6f4ed6599cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7" Jul 31 15:15:47.249954 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:47.249778 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99add764-c1dd-428a-bb04-a6f4ed6599cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7\" (UID: \"99add764-c1dd-428a-bb04-a6f4ed6599cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7" Jul 31 15:15:47.250520 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:47.250487 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99add764-c1dd-428a-bb04-a6f4ed6599cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7\" (UID: \"99add764-c1dd-428a-bb04-a6f4ed6599cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7" Jul 31 15:15:47.259075 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:47.259046 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6cvh\" (UniqueName: \"kubernetes.io/projected/99add764-c1dd-428a-bb04-a6f4ed6599cf-kube-api-access-j6cvh\") pod \"payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7\" (UID: \"99add764-c1dd-428a-bb04-a6f4ed6599cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7" Jul 31 15:15:47.310121 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:47.310084 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7" Jul 31 15:15:47.656995 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:47.656972 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7"] Jul 31 15:15:47.659460 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:15:47.659435 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99add764_c1dd_428a_bb04_a6f4ed6599cf.slice/crio-664ba38d022435db3a694f08596c056503a75c7a41052f8eea0cae89c4f61099 WatchSource:0}: Error finding container 664ba38d022435db3a694f08596c056503a75c7a41052f8eea0cae89c4f61099: Status 404 returned error can't find the container with id 664ba38d022435db3a694f08596c056503a75c7a41052f8eea0cae89c4f61099 Jul 31 15:15:48.191481 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:48.191449 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7" event={"ID":"99add764-c1dd-428a-bb04-a6f4ed6599cf","Type":"ContainerStarted","Data":"9c84923fb16f9c1750b9582b04d3b838715afbf256ea10877000c36c903f1585"} Jul 31 15:15:48.191904 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:48.191500 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7" event={"ID":"99add764-c1dd-428a-bb04-a6f4ed6599cf","Type":"ContainerStarted","Data":"664ba38d022435db3a694f08596c056503a75c7a41052f8eea0cae89c4f61099"} Jul 31 15:15:48.191904 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:48.191571 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7" Jul 31 15:15:48.208333 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:48.208288 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7" podStartSLOduration=2.20824924 podStartE2EDuration="2.20824924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:15:48.207284885 +0000 UTC m=+3800.244107952" watchObservedRunningTime="2026-07-31 15:15:48.20824924 +0000 UTC m=+3800.245072307" Jul 31 15:15:50.969555 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:50.969522 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:15:59.197866 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:15:59.197810 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7" Jul 31 15:16:16.840357 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:16.840321 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7"] Jul 31 15:16:16.840828 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:16.840589 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7" podUID="99add764-c1dd-428a-bb04-a6f4ed6599cf" containerName="payload-processing" containerID="cri-o://9c84923fb16f9c1750b9582b04d3b838715afbf256ea10877000c36c903f1585" gracePeriod=30 Jul 31 15:16:17.098875 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:17.098806 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7" Jul 31 15:16:17.212066 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:17.212032 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6cvh\" (UniqueName: \"kubernetes.io/projected/99add764-c1dd-428a-bb04-a6f4ed6599cf-kube-api-access-j6cvh\") pod \"99add764-c1dd-428a-bb04-a6f4ed6599cf\" (UID: \"99add764-c1dd-428a-bb04-a6f4ed6599cf\") " Jul 31 15:16:17.212235 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:17.212078 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99add764-c1dd-428a-bb04-a6f4ed6599cf-plugins-config-volume\") pod \"99add764-c1dd-428a-bb04-a6f4ed6599cf\" (UID: \"99add764-c1dd-428a-bb04-a6f4ed6599cf\") " Jul 31 15:16:17.212476 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:17.212441 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99add764-c1dd-428a-bb04-a6f4ed6599cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99add764-c1dd-428a-bb04-a6f4ed6599cf" (UID: "99add764-c1dd-428a-bb04-a6f4ed6599cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:16:17.214158 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:17.214136 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99add764-c1dd-428a-bb04-a6f4ed6599cf-kube-api-access-j6cvh" (OuterVolumeSpecName: "kube-api-access-j6cvh") pod "99add764-c1dd-428a-bb04-a6f4ed6599cf" (UID: "99add764-c1dd-428a-bb04-a6f4ed6599cf"). InnerVolumeSpecName "kube-api-access-j6cvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:16:17.313732 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:17.313698 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6cvh\" (UniqueName: \"kubernetes.io/projected/99add764-c1dd-428a-bb04-a6f4ed6599cf-kube-api-access-j6cvh\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:16:17.313732 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:17.313730 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99add764-c1dd-428a-bb04-a6f4ed6599cf-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:16:17.328191 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:17.328160 2577 generic.go:356] "Generic (PLEG): container finished" podID="99add764-c1dd-428a-bb04-a6f4ed6599cf" containerID="9c84923fb16f9c1750b9582b04d3b838715afbf256ea10877000c36c903f1585" exitCode=0 Jul 31 15:16:17.328311 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:17.328204 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7" Jul 31 15:16:17.328311 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:17.328226 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7" event={"ID":"99add764-c1dd-428a-bb04-a6f4ed6599cf","Type":"ContainerDied","Data":"9c84923fb16f9c1750b9582b04d3b838715afbf256ea10877000c36c903f1585"} Jul 31 15:16:17.328311 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:17.328280 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7" event={"ID":"99add764-c1dd-428a-bb04-a6f4ed6599cf","Type":"ContainerDied","Data":"664ba38d022435db3a694f08596c056503a75c7a41052f8eea0cae89c4f61099"} Jul 31 15:16:17.328311 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:17.328310 2577 scope.go:117] "RemoveContainer" containerID="9c84923fb16f9c1750b9582b04d3b838715afbf256ea10877000c36c903f1585" Jul 31 15:16:17.337603 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:17.337583 2577 scope.go:117] "RemoveContainer" containerID="9c84923fb16f9c1750b9582b04d3b838715afbf256ea10877000c36c903f1585" Jul 31 15:16:17.337877 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:16:17.337829 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c84923fb16f9c1750b9582b04d3b838715afbf256ea10877000c36c903f1585\": container with ID starting with 9c84923fb16f9c1750b9582b04d3b838715afbf256ea10877000c36c903f1585 not found: ID does not exist" containerID="9c84923fb16f9c1750b9582b04d3b838715afbf256ea10877000c36c903f1585" Jul 31 15:16:17.337965 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:17.337882 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c84923fb16f9c1750b9582b04d3b838715afbf256ea10877000c36c903f1585"} err="failed to get container status \"9c84923fb16f9c1750b9582b04d3b838715afbf256ea10877000c36c903f1585\": rpc error: code = NotFound desc = could not find container \"9c84923fb16f9c1750b9582b04d3b838715afbf256ea10877000c36c903f1585\": container with ID starting with 9c84923fb16f9c1750b9582b04d3b838715afbf256ea10877000c36c903f1585 not found: ID does not exist" Jul 31 15:16:17.349437 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:17.349387 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7"] Jul 31 15:16:17.354547 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:17.354524 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-39573e-57d74567b4-nbvp7"] Jul 31 15:16:18.429958 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:18.429924 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99add764-c1dd-428a-bb04-a6f4ed6599cf" path="/var/lib/kubelet/pods/99add764-c1dd-428a-bb04-a6f4ed6599cf/volumes" Jul 31 15:16:19.878578 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:19.878541 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:16:40.578025 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:40.577991 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:16:42.709488 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:42.709438 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh"] Jul 31 15:16:42.709993 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:42.709693 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh" podUID="4feaae1a-54a5-4fc7-843e-d0c82255c13a" containerName="payload-pre-processing" containerID="cri-o://8e55bbf5ac148313dfe0094364ff2dce25530dee262e7587d8eb6a240dffae76" gracePeriod=30 Jul 31 15:16:42.976970 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:42.976899 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh" Jul 31 15:16:43.044259 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:43.044221 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94wwg\" (UniqueName: \"kubernetes.io/projected/4feaae1a-54a5-4fc7-843e-d0c82255c13a-kube-api-access-94wwg\") pod \"4feaae1a-54a5-4fc7-843e-d0c82255c13a\" (UID: \"4feaae1a-54a5-4fc7-843e-d0c82255c13a\") " Jul 31 15:16:43.044429 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:43.044285 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4feaae1a-54a5-4fc7-843e-d0c82255c13a-plugins-config-volume\") pod \"4feaae1a-54a5-4fc7-843e-d0c82255c13a\" (UID: \"4feaae1a-54a5-4fc7-843e-d0c82255c13a\") " Jul 31 15:16:43.044660 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:43.044636 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4feaae1a-54a5-4fc7-843e-d0c82255c13a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4feaae1a-54a5-4fc7-843e-d0c82255c13a" (UID: "4feaae1a-54a5-4fc7-843e-d0c82255c13a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:16:43.046421 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:43.046397 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4feaae1a-54a5-4fc7-843e-d0c82255c13a-kube-api-access-94wwg" (OuterVolumeSpecName: "kube-api-access-94wwg") pod "4feaae1a-54a5-4fc7-843e-d0c82255c13a" (UID: "4feaae1a-54a5-4fc7-843e-d0c82255c13a"). InnerVolumeSpecName "kube-api-access-94wwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:16:43.145163 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:43.145132 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4feaae1a-54a5-4fc7-843e-d0c82255c13a-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:16:43.145163 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:43.145162 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94wwg\" (UniqueName: \"kubernetes.io/projected/4feaae1a-54a5-4fc7-843e-d0c82255c13a-kube-api-access-94wwg\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:16:43.452046 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:43.452018 2577 generic.go:356] "Generic (PLEG): container finished" podID="4feaae1a-54a5-4fc7-843e-d0c82255c13a" containerID="8e55bbf5ac148313dfe0094364ff2dce25530dee262e7587d8eb6a240dffae76" exitCode=0 Jul 31 15:16:43.452216 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:43.452064 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh" Jul 31 15:16:43.452216 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:43.452107 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh" event={"ID":"4feaae1a-54a5-4fc7-843e-d0c82255c13a","Type":"ContainerDied","Data":"8e55bbf5ac148313dfe0094364ff2dce25530dee262e7587d8eb6a240dffae76"} Jul 31 15:16:43.452216 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:43.452165 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh" event={"ID":"4feaae1a-54a5-4fc7-843e-d0c82255c13a","Type":"ContainerDied","Data":"bd86feb37fc19dbad7c53a177fd590deac6a719e750c31341e4f67c90dfc8d96"} Jul 31 15:16:43.452216 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:43.452194 2577 scope.go:117] "RemoveContainer" containerID="8e55bbf5ac148313dfe0094364ff2dce25530dee262e7587d8eb6a240dffae76" Jul 31 15:16:43.460828 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:43.460812 2577 scope.go:117] "RemoveContainer" containerID="8e55bbf5ac148313dfe0094364ff2dce25530dee262e7587d8eb6a240dffae76" Jul 31 15:16:43.461126 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:16:43.461096 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e55bbf5ac148313dfe0094364ff2dce25530dee262e7587d8eb6a240dffae76\": container with ID starting with 8e55bbf5ac148313dfe0094364ff2dce25530dee262e7587d8eb6a240dffae76 not found: ID does not exist" containerID="8e55bbf5ac148313dfe0094364ff2dce25530dee262e7587d8eb6a240dffae76" Jul 31 15:16:43.461197 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:43.461125 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e55bbf5ac148313dfe0094364ff2dce25530dee262e7587d8eb6a240dffae76"} err="failed to get container status \"8e55bbf5ac148313dfe0094364ff2dce25530dee262e7587d8eb6a240dffae76\": rpc error: code = NotFound desc = could not find container \"8e55bbf5ac148313dfe0094364ff2dce25530dee262e7587d8eb6a240dffae76\": container with ID starting with 8e55bbf5ac148313dfe0094364ff2dce25530dee262e7587d8eb6a240dffae76 not found: ID does not exist" Jul 31 15:16:43.471196 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:43.471175 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh"] Jul 31 15:16:43.476781 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:43.476759 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-58af04-65f8577468-hdjjh"] Jul 31 15:16:44.430389 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:44.430357 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4feaae1a-54a5-4fc7-843e-d0c82255c13a" path="/var/lib/kubelet/pods/4feaae1a-54a5-4fc7-843e-d0c82255c13a/volumes" Jul 31 15:16:50.081384 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:16:50.081347 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:17:10.597711 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:10.597678 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v"] Jul 31 15:17:10.598126 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:10.597970 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" podUID="de459d3d-e0b4-4810-a31b-666efc74cae2" containerName="istio-proxy" containerID="cri-o://d9f687152004c4fad4c1081426d2bf593f92324d3e161456928a0cebec88f382" gracePeriod=30 Jul 31 15:17:15.840194 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.840169 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:17:15.939227 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.939200 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-podinfo\") pod \"de459d3d-e0b4-4810-a31b-666efc74cae2\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " Jul 31 15:17:15.939408 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.939248 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-envoy\") pod \"de459d3d-e0b4-4810-a31b-666efc74cae2\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " Jul 31 15:17:15.939408 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.939285 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-credential-socket\") pod \"de459d3d-e0b4-4810-a31b-666efc74cae2\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " Jul 31 15:17:15.939408 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.939340 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-data\") pod \"de459d3d-e0b4-4810-a31b-666efc74cae2\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " Jul 31 15:17:15.939408 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.939371 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-token\") pod \"de459d3d-e0b4-4810-a31b-666efc74cae2\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " Jul 31 15:17:15.939628 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.939417 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfblb\" (UniqueName: \"kubernetes.io/projected/de459d3d-e0b4-4810-a31b-666efc74cae2-kube-api-access-wfblb\") pod \"de459d3d-e0b4-4810-a31b-666efc74cae2\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " Jul 31 15:17:15.939628 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.939438 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-workload-socket\") pod \"de459d3d-e0b4-4810-a31b-666efc74cae2\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " Jul 31 15:17:15.939628 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.939475 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-workload-certs\") pod \"de459d3d-e0b4-4810-a31b-666efc74cae2\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " Jul 31 15:17:15.939628 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.939521 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de459d3d-e0b4-4810-a31b-666efc74cae2-istiod-ca-cert\") pod \"de459d3d-e0b4-4810-a31b-666efc74cae2\" (UID: \"de459d3d-e0b4-4810-a31b-666efc74cae2\") " Jul 31 15:17:15.939628 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.939614 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "de459d3d-e0b4-4810-a31b-666efc74cae2" (UID: "de459d3d-e0b4-4810-a31b-666efc74cae2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:17:15.939919 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.939643 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-data" (OuterVolumeSpecName: "istio-data") pod "de459d3d-e0b4-4810-a31b-666efc74cae2" (UID: "de459d3d-e0b4-4810-a31b-666efc74cae2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:17:15.939919 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.939814 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "de459d3d-e0b4-4810-a31b-666efc74cae2" (UID: "de459d3d-e0b4-4810-a31b-666efc74cae2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:17:15.939919 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.939892 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-credential-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:15.939919 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.939913 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-data\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:15.940128 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.939993 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de459d3d-e0b4-4810-a31b-666efc74cae2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "de459d3d-e0b4-4810-a31b-666efc74cae2" (UID: "de459d3d-e0b4-4810-a31b-666efc74cae2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:17:15.940128 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.940012 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "de459d3d-e0b4-4810-a31b-666efc74cae2" (UID: "de459d3d-e0b4-4810-a31b-666efc74cae2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:17:15.941584 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.941562 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "de459d3d-e0b4-4810-a31b-666efc74cae2" (UID: "de459d3d-e0b4-4810-a31b-666efc74cae2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:17:15.941701 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.941585 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-token" (OuterVolumeSpecName: "istio-token") pod "de459d3d-e0b4-4810-a31b-666efc74cae2" (UID: "de459d3d-e0b4-4810-a31b-666efc74cae2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:17:15.941701 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.941596 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "de459d3d-e0b4-4810-a31b-666efc74cae2" (UID: "de459d3d-e0b4-4810-a31b-666efc74cae2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 15:17:15.941701 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:15.941624 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de459d3d-e0b4-4810-a31b-666efc74cae2-kube-api-access-wfblb" (OuterVolumeSpecName: "kube-api-access-wfblb") pod "de459d3d-e0b4-4810-a31b-666efc74cae2" (UID: "de459d3d-e0b4-4810-a31b-666efc74cae2"). InnerVolumeSpecName "kube-api-access-wfblb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:17:16.040960 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.040934 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-workload-certs\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:16.040960 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.040959 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de459d3d-e0b4-4810-a31b-666efc74cae2-istiod-ca-cert\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:16.041209 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.040969 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-podinfo\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:16.041209 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.040977 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-envoy\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:16.041209 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.040985 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de459d3d-e0b4-4810-a31b-666efc74cae2-istio-token\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:16.041209 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.040994 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfblb\" (UniqueName: \"kubernetes.io/projected/de459d3d-e0b4-4810-a31b-666efc74cae2-kube-api-access-wfblb\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:16.041209 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.041003 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de459d3d-e0b4-4810-a31b-666efc74cae2-workload-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:16.606703 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.606661 2577 generic.go:356] "Generic (PLEG): container finished" podID="de459d3d-e0b4-4810-a31b-666efc74cae2" containerID="d9f687152004c4fad4c1081426d2bf593f92324d3e161456928a0cebec88f382" exitCode=0 Jul 31 15:17:16.606904 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.606721 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" Jul 31 15:17:16.606904 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.606744 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" event={"ID":"de459d3d-e0b4-4810-a31b-666efc74cae2","Type":"ContainerDied","Data":"d9f687152004c4fad4c1081426d2bf593f92324d3e161456928a0cebec88f382"} Jul 31 15:17:16.606904 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.606804 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v" event={"ID":"de459d3d-e0b4-4810-a31b-666efc74cae2","Type":"ContainerDied","Data":"35f3db4f4a57d9738f33116dc27d180143428b6e63d6cf0a52a3a4c3bdd71b55"} Jul 31 15:17:16.606904 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.606829 2577 scope.go:117] "RemoveContainer" containerID="d9f687152004c4fad4c1081426d2bf593f92324d3e161456928a0cebec88f382" Jul 31 15:17:16.615535 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.615520 2577 scope.go:117] "RemoveContainer" containerID="dfe7e16bad0561d3b01f6e62d664f5e6215ca011fb31f7199e094b0fe07eaae5" Jul 31 15:17:16.623817 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.623594 2577 scope.go:117] "RemoveContainer" containerID="d9f687152004c4fad4c1081426d2bf593f92324d3e161456928a0cebec88f382" Jul 31 15:17:16.623912 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:17:16.623870 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9f687152004c4fad4c1081426d2bf593f92324d3e161456928a0cebec88f382\": container with ID starting with d9f687152004c4fad4c1081426d2bf593f92324d3e161456928a0cebec88f382 not found: ID does not exist" containerID="d9f687152004c4fad4c1081426d2bf593f92324d3e161456928a0cebec88f382" Jul 31 15:17:16.623912 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.623891 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f687152004c4fad4c1081426d2bf593f92324d3e161456928a0cebec88f382"} err="failed to get container status \"d9f687152004c4fad4c1081426d2bf593f92324d3e161456928a0cebec88f382\": rpc error: code = NotFound desc = could not find container \"d9f687152004c4fad4c1081426d2bf593f92324d3e161456928a0cebec88f382\": container with ID starting with d9f687152004c4fad4c1081426d2bf593f92324d3e161456928a0cebec88f382 not found: ID does not exist" Jul 31 15:17:16.623912 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.623908 2577 scope.go:117] "RemoveContainer" containerID="dfe7e16bad0561d3b01f6e62d664f5e6215ca011fb31f7199e094b0fe07eaae5" Jul 31 15:17:16.624149 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:17:16.624130 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe7e16bad0561d3b01f6e62d664f5e6215ca011fb31f7199e094b0fe07eaae5\": container with ID starting with dfe7e16bad0561d3b01f6e62d664f5e6215ca011fb31f7199e094b0fe07eaae5 not found: ID does not exist" containerID="dfe7e16bad0561d3b01f6e62d664f5e6215ca011fb31f7199e094b0fe07eaae5" Jul 31 15:17:16.624193 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.624155 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe7e16bad0561d3b01f6e62d664f5e6215ca011fb31f7199e094b0fe07eaae5"} err="failed to get container status \"dfe7e16bad0561d3b01f6e62d664f5e6215ca011fb31f7199e094b0fe07eaae5\": rpc error: code = NotFound desc = could not find container \"dfe7e16bad0561d3b01f6e62d664f5e6215ca011fb31f7199e094b0fe07eaae5\": container with ID starting with dfe7e16bad0561d3b01f6e62d664f5e6215ca011fb31f7199e094b0fe07eaae5 not found: ID does not exist" Jul 31 15:17:16.624263 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.624244 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v"] Jul 31 15:17:16.628486 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:16.628466 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-58af04-openshift-default-d6d59f9fd-m5v2v"] Jul 31 15:17:18.430053 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:18.430018 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de459d3d-e0b4-4810-a31b-666efc74cae2" path="/var/lib/kubelet/pods/de459d3d-e0b4-4810-a31b-666efc74cae2/volumes" Jul 31 15:17:21.411959 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.411923 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs"] Jul 31 15:17:21.412556 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.412174 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs" podUID="3b4a38ac-d1ec-439e-aa0d-c498508794ee" containerName="payload-processing" containerID="cri-o://00cd8611375c4752ba6d75895e53be2e42350433b7611da85ca790bd1ea47432" gracePeriod=30 Jul 31 15:17:21.418972 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.418942 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd"] Jul 31 15:17:21.419212 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.419164 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd" podUID="10dd78e5-a052-45f1-805c-8f7a9a1db0fa" containerName="payload-pre-processing" containerID="cri-o://73dff7a42bc376eda4c7650ff8461fcf58ba71bef201d770a9798a20ef42ae10" gracePeriod=30 Jul 31 15:17:21.633231 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.632550 2577 generic.go:356] "Generic (PLEG): container finished" podID="3b4a38ac-d1ec-439e-aa0d-c498508794ee" containerID="00cd8611375c4752ba6d75895e53be2e42350433b7611da85ca790bd1ea47432" exitCode=0 Jul 31 15:17:21.633231 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.632680 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs" event={"ID":"3b4a38ac-d1ec-439e-aa0d-c498508794ee","Type":"ContainerDied","Data":"00cd8611375c4752ba6d75895e53be2e42350433b7611da85ca790bd1ea47432"} Jul 31 15:17:21.634893 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.634863 2577 generic.go:356] "Generic (PLEG): container finished" podID="10dd78e5-a052-45f1-805c-8f7a9a1db0fa" containerID="73dff7a42bc376eda4c7650ff8461fcf58ba71bef201d770a9798a20ef42ae10" exitCode=0 Jul 31 15:17:21.635022 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.634920 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd" event={"ID":"10dd78e5-a052-45f1-805c-8f7a9a1db0fa","Type":"ContainerDied","Data":"73dff7a42bc376eda4c7650ff8461fcf58ba71bef201d770a9798a20ef42ae10"} Jul 31 15:17:21.682940 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.682913 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs" Jul 31 15:17:21.710939 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.710911 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd" Jul 31 15:17:21.791088 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.791052 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4a38ac-d1ec-439e-aa0d-c498508794ee-plugins-config-volume\") pod \"3b4a38ac-d1ec-439e-aa0d-c498508794ee\" (UID: \"3b4a38ac-d1ec-439e-aa0d-c498508794ee\") " Jul 31 15:17:21.791310 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.791188 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwt9x\" (UniqueName: \"kubernetes.io/projected/3b4a38ac-d1ec-439e-aa0d-c498508794ee-kube-api-access-zwt9x\") pod \"3b4a38ac-d1ec-439e-aa0d-c498508794ee\" (UID: \"3b4a38ac-d1ec-439e-aa0d-c498508794ee\") " Jul 31 15:17:21.791468 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.791444 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b4a38ac-d1ec-439e-aa0d-c498508794ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b4a38ac-d1ec-439e-aa0d-c498508794ee" (UID: "3b4a38ac-d1ec-439e-aa0d-c498508794ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:17:21.793317 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.793298 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4a38ac-d1ec-439e-aa0d-c498508794ee-kube-api-access-zwt9x" (OuterVolumeSpecName: "kube-api-access-zwt9x") pod "3b4a38ac-d1ec-439e-aa0d-c498508794ee" (UID: "3b4a38ac-d1ec-439e-aa0d-c498508794ee"). InnerVolumeSpecName "kube-api-access-zwt9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:17:21.892153 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.892119 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10dd78e5-a052-45f1-805c-8f7a9a1db0fa-plugins-config-volume\") pod \"10dd78e5-a052-45f1-805c-8f7a9a1db0fa\" (UID: \"10dd78e5-a052-45f1-805c-8f7a9a1db0fa\") " Jul 31 15:17:21.892319 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.892168 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxl44\" (UniqueName: \"kubernetes.io/projected/10dd78e5-a052-45f1-805c-8f7a9a1db0fa-kube-api-access-cxl44\") pod \"10dd78e5-a052-45f1-805c-8f7a9a1db0fa\" (UID: \"10dd78e5-a052-45f1-805c-8f7a9a1db0fa\") " Jul 31 15:17:21.892410 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.892381 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwt9x\" (UniqueName: \"kubernetes.io/projected/3b4a38ac-d1ec-439e-aa0d-c498508794ee-kube-api-access-zwt9x\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:21.892410 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.892392 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4a38ac-d1ec-439e-aa0d-c498508794ee-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:21.892519 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.892500 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10dd78e5-a052-45f1-805c-8f7a9a1db0fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "10dd78e5-a052-45f1-805c-8f7a9a1db0fa" (UID: "10dd78e5-a052-45f1-805c-8f7a9a1db0fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:17:21.894256 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.894233 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10dd78e5-a052-45f1-805c-8f7a9a1db0fa-kube-api-access-cxl44" (OuterVolumeSpecName: "kube-api-access-cxl44") pod "10dd78e5-a052-45f1-805c-8f7a9a1db0fa" (UID: "10dd78e5-a052-45f1-805c-8f7a9a1db0fa"). InnerVolumeSpecName "kube-api-access-cxl44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:17:21.993478 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.993386 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10dd78e5-a052-45f1-805c-8f7a9a1db0fa-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:21.993478 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:21.993418 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxl44\" (UniqueName: \"kubernetes.io/projected/10dd78e5-a052-45f1-805c-8f7a9a1db0fa-kube-api-access-cxl44\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:22.641562 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:22.641536 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd" Jul 31 15:17:22.642079 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:22.641558 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd" event={"ID":"10dd78e5-a052-45f1-805c-8f7a9a1db0fa","Type":"ContainerDied","Data":"d25230325e25d7af9b034e7fea42b521718ffa10ba465cf55c5b340ff842974a"} Jul 31 15:17:22.642079 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:22.641627 2577 scope.go:117] "RemoveContainer" containerID="73dff7a42bc376eda4c7650ff8461fcf58ba71bef201d770a9798a20ef42ae10" Jul 31 15:17:22.643008 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:22.642923 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs" event={"ID":"3b4a38ac-d1ec-439e-aa0d-c498508794ee","Type":"ContainerDied","Data":"a3b8d1ba90aee2fa1bab37dd46c079e1dd6611a8a39a2a595682647900879b62"} Jul 31 15:17:22.643008 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:22.642956 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs" Jul 31 15:17:22.650555 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:22.650536 2577 scope.go:117] "RemoveContainer" containerID="00cd8611375c4752ba6d75895e53be2e42350433b7611da85ca790bd1ea47432" Jul 31 15:17:22.659670 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:22.659649 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd"] Jul 31 15:17:22.663765 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:22.663741 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c9d21-65c75fc668-bkfgd"] Jul 31 15:17:22.674174 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:22.674147 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs"] Jul 31 15:17:22.678883 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:22.678860 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c9d21-bf597c848-6bnrs"] Jul 31 15:17:23.684342 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:23.684309 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:17:24.429635 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:24.429600 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10dd78e5-a052-45f1-805c-8f7a9a1db0fa" path="/var/lib/kubelet/pods/10dd78e5-a052-45f1-805c-8f7a9a1db0fa/volumes" Jul 31 15:17:24.430008 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:24.429994 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b4a38ac-d1ec-439e-aa0d-c498508794ee" path="/var/lib/kubelet/pods/3b4a38ac-d1ec-439e-aa0d-c498508794ee/volumes" Jul 31 15:17:39.176255 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.176216 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4"] Jul 31 15:17:39.176744 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.176515 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4" podUID="89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f" containerName="payload-processing" containerID="cri-o://a376b967aaef9e9f34620be7dd7f80c29892ffe55b238bedac0256534195d325" gracePeriod=30 Jul 31 15:17:39.181096 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.181071 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd"] Jul 31 15:17:39.181366 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.181321 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd" podUID="06e3de25-5249-4bfd-bf75-a0c0b013b660" containerName="payload-pre-processing" containerID="cri-o://bf43374897623cc4a6416a22e68ffbbe94d86f3a46b47fe76c9cae42f5e84ac6" gracePeriod=30 Jul 31 15:17:39.464105 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.464072 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4" Jul 31 15:17:39.466749 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.466730 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd" Jul 31 15:17:39.531557 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.531532 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69x27\" (UniqueName: \"kubernetes.io/projected/06e3de25-5249-4bfd-bf75-a0c0b013b660-kube-api-access-69x27\") pod \"06e3de25-5249-4bfd-bf75-a0c0b013b660\" (UID: \"06e3de25-5249-4bfd-bf75-a0c0b013b660\") " Jul 31 15:17:39.531690 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.531586 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e3de25-5249-4bfd-bf75-a0c0b013b660-plugins-config-volume\") pod \"06e3de25-5249-4bfd-bf75-a0c0b013b660\" (UID: \"06e3de25-5249-4bfd-bf75-a0c0b013b660\") " Jul 31 15:17:39.531690 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.531670 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92nvd\" (UniqueName: \"kubernetes.io/projected/89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f-kube-api-access-92nvd\") pod \"89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f\" (UID: \"89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f\") " Jul 31 15:17:39.531769 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.531695 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f-plugins-config-volume\") pod \"89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f\" (UID: \"89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f\") " Jul 31 15:17:39.532065 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.531972 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e3de25-5249-4bfd-bf75-a0c0b013b660-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "06e3de25-5249-4bfd-bf75-a0c0b013b660" (UID: "06e3de25-5249-4bfd-bf75-a0c0b013b660"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:17:39.532173 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.532124 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f" (UID: "89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:17:39.533659 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.533639 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e3de25-5249-4bfd-bf75-a0c0b013b660-kube-api-access-69x27" (OuterVolumeSpecName: "kube-api-access-69x27") pod "06e3de25-5249-4bfd-bf75-a0c0b013b660" (UID: "06e3de25-5249-4bfd-bf75-a0c0b013b660"). InnerVolumeSpecName "kube-api-access-69x27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:17:39.533717 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.533664 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f-kube-api-access-92nvd" (OuterVolumeSpecName: "kube-api-access-92nvd") pod "89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f" (UID: "89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f"). InnerVolumeSpecName "kube-api-access-92nvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:17:39.632253 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.632213 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92nvd\" (UniqueName: \"kubernetes.io/projected/89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f-kube-api-access-92nvd\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:39.632253 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.632249 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:39.632253 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.632259 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69x27\" (UniqueName: \"kubernetes.io/projected/06e3de25-5249-4bfd-bf75-a0c0b013b660-kube-api-access-69x27\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:39.632444 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.632269 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e3de25-5249-4bfd-bf75-a0c0b013b660-plugins-config-volume\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:39.720745 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.720667 2577 generic.go:356] "Generic (PLEG): container finished" podID="89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f" containerID="a376b967aaef9e9f34620be7dd7f80c29892ffe55b238bedac0256534195d325" exitCode=0 Jul 31 15:17:39.720745 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.720708 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4" Jul 31 15:17:39.720967 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.720749 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4" event={"ID":"89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f","Type":"ContainerDied","Data":"a376b967aaef9e9f34620be7dd7f80c29892ffe55b238bedac0256534195d325"} Jul 31 15:17:39.720967 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.720803 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4" event={"ID":"89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f","Type":"ContainerDied","Data":"2a9c2b66101f0a2e50f3e847af51f5c07b894bfa8c760693191f23b9aeb0df9d"} Jul 31 15:17:39.720967 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.720836 2577 scope.go:117] "RemoveContainer" containerID="a376b967aaef9e9f34620be7dd7f80c29892ffe55b238bedac0256534195d325" Jul 31 15:17:39.722022 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.722003 2577 generic.go:356] "Generic (PLEG): container finished" podID="06e3de25-5249-4bfd-bf75-a0c0b013b660" containerID="bf43374897623cc4a6416a22e68ffbbe94d86f3a46b47fe76c9cae42f5e84ac6" exitCode=0 Jul 31 15:17:39.722120 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.722043 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd" Jul 31 15:17:39.722120 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.722077 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd" event={"ID":"06e3de25-5249-4bfd-bf75-a0c0b013b660","Type":"ContainerDied","Data":"bf43374897623cc4a6416a22e68ffbbe94d86f3a46b47fe76c9cae42f5e84ac6"} Jul 31 15:17:39.722120 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.722117 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd" event={"ID":"06e3de25-5249-4bfd-bf75-a0c0b013b660","Type":"ContainerDied","Data":"b0779eb10c4cbf216c5e402ee90a7a9bcd8ffcdd02c8dfb4bd4250a591738685"} Jul 31 15:17:39.729071 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.729052 2577 scope.go:117] "RemoveContainer" containerID="a376b967aaef9e9f34620be7dd7f80c29892ffe55b238bedac0256534195d325" Jul 31 15:17:39.729427 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:17:39.729407 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a376b967aaef9e9f34620be7dd7f80c29892ffe55b238bedac0256534195d325\": container with ID starting with a376b967aaef9e9f34620be7dd7f80c29892ffe55b238bedac0256534195d325 not found: ID does not exist" containerID="a376b967aaef9e9f34620be7dd7f80c29892ffe55b238bedac0256534195d325" Jul 31 15:17:39.729514 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.729440 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a376b967aaef9e9f34620be7dd7f80c29892ffe55b238bedac0256534195d325"} err="failed to get container status \"a376b967aaef9e9f34620be7dd7f80c29892ffe55b238bedac0256534195d325\": rpc error: code = NotFound desc = could not find container \"a376b967aaef9e9f34620be7dd7f80c29892ffe55b238bedac0256534195d325\": container with ID starting with a376b967aaef9e9f34620be7dd7f80c29892ffe55b238bedac0256534195d325 not found: ID does not exist" Jul 31 15:17:39.729514 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.729462 2577 scope.go:117] "RemoveContainer" containerID="bf43374897623cc4a6416a22e68ffbbe94d86f3a46b47fe76c9cae42f5e84ac6" Jul 31 15:17:39.737117 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.737103 2577 scope.go:117] "RemoveContainer" containerID="bf43374897623cc4a6416a22e68ffbbe94d86f3a46b47fe76c9cae42f5e84ac6" Jul 31 15:17:39.737384 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:17:39.737362 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf43374897623cc4a6416a22e68ffbbe94d86f3a46b47fe76c9cae42f5e84ac6\": container with ID starting with bf43374897623cc4a6416a22e68ffbbe94d86f3a46b47fe76c9cae42f5e84ac6 not found: ID does not exist" containerID="bf43374897623cc4a6416a22e68ffbbe94d86f3a46b47fe76c9cae42f5e84ac6" Jul 31 15:17:39.737460 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.737388 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf43374897623cc4a6416a22e68ffbbe94d86f3a46b47fe76c9cae42f5e84ac6"} err="failed to get container status \"bf43374897623cc4a6416a22e68ffbbe94d86f3a46b47fe76c9cae42f5e84ac6\": rpc error: code = NotFound desc = could not find container \"bf43374897623cc4a6416a22e68ffbbe94d86f3a46b47fe76c9cae42f5e84ac6\": container with ID starting with bf43374897623cc4a6416a22e68ffbbe94d86f3a46b47fe76c9cae42f5e84ac6 not found: ID does not exist" Jul 31 15:17:39.743978 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.743957 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4"] Jul 31 15:17:39.747679 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.747660 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-9966b2-65c745bbdb-dfdg4"] Jul 31 15:17:39.756910 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.756872 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd"] Jul 31 15:17:39.762801 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:39.762782 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-9966b2-57c97f67b-mmsbd"] Jul 31 15:17:40.429633 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:40.429600 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06e3de25-5249-4bfd-bf75-a0c0b013b660" path="/var/lib/kubelet/pods/06e3de25-5249-4bfd-bf75-a0c0b013b660/volumes" Jul 31 15:17:40.429994 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:40.429974 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f" path="/var/lib/kubelet/pods/89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f/volumes" Jul 31 15:17:41.376217 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:41.376181 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:17:45.894818 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:45.894783 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw"] Jul 31 15:17:45.895289 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:45.895068 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" podUID="8a78760f-ff69-4f1c-800f-8c9cac414b13" containerName="istio-proxy" containerID="cri-o://d92bf2f2e03a81889e37d5367d4e800bdadf064d3bbde13e02b40b0ce41d1ca2" gracePeriod=30 Jul 31 15:17:51.141222 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.141190 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:17:51.229162 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229087 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-workload-certs\") pod \"8a78760f-ff69-4f1c-800f-8c9cac414b13\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " Jul 31 15:17:51.229162 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229127 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-podinfo\") pod \"8a78760f-ff69-4f1c-800f-8c9cac414b13\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " Jul 31 15:17:51.229162 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229147 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-credential-socket\") pod \"8a78760f-ff69-4f1c-800f-8c9cac414b13\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " Jul 31 15:17:51.229414 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229171 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a78760f-ff69-4f1c-800f-8c9cac414b13-istiod-ca-cert\") pod \"8a78760f-ff69-4f1c-800f-8c9cac414b13\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " Jul 31 15:17:51.229414 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229219 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-data\") pod \"8a78760f-ff69-4f1c-800f-8c9cac414b13\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " Jul 31 15:17:51.229414 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229248 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znlzb\" (UniqueName: \"kubernetes.io/projected/8a78760f-ff69-4f1c-800f-8c9cac414b13-kube-api-access-znlzb\") pod \"8a78760f-ff69-4f1c-800f-8c9cac414b13\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " Jul 31 15:17:51.229414 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229274 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-token\") pod \"8a78760f-ff69-4f1c-800f-8c9cac414b13\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " Jul 31 15:17:51.229414 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229311 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-envoy\") pod \"8a78760f-ff69-4f1c-800f-8c9cac414b13\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " Jul 31 15:17:51.229414 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229399 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8a78760f-ff69-4f1c-800f-8c9cac414b13" (UID: "8a78760f-ff69-4f1c-800f-8c9cac414b13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:17:51.229719 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229418 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8a78760f-ff69-4f1c-800f-8c9cac414b13" (UID: "8a78760f-ff69-4f1c-800f-8c9cac414b13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:17:51.229719 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229468 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-workload-socket\") pod \"8a78760f-ff69-4f1c-800f-8c9cac414b13\" (UID: \"8a78760f-ff69-4f1c-800f-8c9cac414b13\") " Jul 31 15:17:51.229719 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229535 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-data" (OuterVolumeSpecName: "istio-data") pod "8a78760f-ff69-4f1c-800f-8c9cac414b13" (UID: "8a78760f-ff69-4f1c-800f-8c9cac414b13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:17:51.229719 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229638 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a78760f-ff69-4f1c-800f-8c9cac414b13-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8a78760f-ff69-4f1c-800f-8c9cac414b13" (UID: "8a78760f-ff69-4f1c-800f-8c9cac414b13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:17:51.229719 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229707 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8a78760f-ff69-4f1c-800f-8c9cac414b13" (UID: "8a78760f-ff69-4f1c-800f-8c9cac414b13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:17:51.229996 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229906 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-workload-certs\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:51.229996 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229929 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-credential-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:51.229996 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229943 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a78760f-ff69-4f1c-800f-8c9cac414b13-istiod-ca-cert\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:51.229996 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229957 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-data\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:51.229996 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.229971 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-workload-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:51.231388 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.231364 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8a78760f-ff69-4f1c-800f-8c9cac414b13" (UID: "8a78760f-ff69-4f1c-800f-8c9cac414b13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:17:51.231482 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.231422 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a78760f-ff69-4f1c-800f-8c9cac414b13-kube-api-access-znlzb" (OuterVolumeSpecName: "kube-api-access-znlzb") pod "8a78760f-ff69-4f1c-800f-8c9cac414b13" (UID: "8a78760f-ff69-4f1c-800f-8c9cac414b13"). InnerVolumeSpecName "kube-api-access-znlzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:17:51.231721 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.231699 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8a78760f-ff69-4f1c-800f-8c9cac414b13" (UID: "8a78760f-ff69-4f1c-800f-8c9cac414b13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 15:17:51.231816 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.231716 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-token" (OuterVolumeSpecName: "istio-token") pod "8a78760f-ff69-4f1c-800f-8c9cac414b13" (UID: "8a78760f-ff69-4f1c-800f-8c9cac414b13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:17:51.331268 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.331233 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znlzb\" (UniqueName: \"kubernetes.io/projected/8a78760f-ff69-4f1c-800f-8c9cac414b13-kube-api-access-znlzb\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:51.331268 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.331264 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-token\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:51.331268 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.331274 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-envoy\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:51.331500 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.331284 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a78760f-ff69-4f1c-800f-8c9cac414b13-istio-podinfo\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:17:51.777819 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.777779 2577 generic.go:356] "Generic (PLEG): container finished" podID="8a78760f-ff69-4f1c-800f-8c9cac414b13" containerID="d92bf2f2e03a81889e37d5367d4e800bdadf064d3bbde13e02b40b0ce41d1ca2" exitCode=0 Jul 31 15:17:51.778037 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.777853 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" Jul 31 15:17:51.778037 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.777869 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" event={"ID":"8a78760f-ff69-4f1c-800f-8c9cac414b13","Type":"ContainerDied","Data":"d92bf2f2e03a81889e37d5367d4e800bdadf064d3bbde13e02b40b0ce41d1ca2"} Jul 31 15:17:51.778037 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.777922 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw" event={"ID":"8a78760f-ff69-4f1c-800f-8c9cac414b13","Type":"ContainerDied","Data":"98540210cc015bd362c963f788535d20f6d7ab9b7e5a8524cd0d949b63eccdec"} Jul 31 15:17:51.778037 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.777956 2577 scope.go:117] "RemoveContainer" containerID="d92bf2f2e03a81889e37d5367d4e800bdadf064d3bbde13e02b40b0ce41d1ca2" Jul 31 15:17:51.786589 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.786569 2577 scope.go:117] "RemoveContainer" containerID="0a4c6fc43b63230a2d770aa53e664e5f99aa4c7b17a9bec405243e53ebaa9a0a" Jul 31 15:17:51.793935 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.793917 2577 scope.go:117] "RemoveContainer" containerID="d92bf2f2e03a81889e37d5367d4e800bdadf064d3bbde13e02b40b0ce41d1ca2" Jul 31 15:17:51.794190 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:17:51.794171 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92bf2f2e03a81889e37d5367d4e800bdadf064d3bbde13e02b40b0ce41d1ca2\": container with ID starting with d92bf2f2e03a81889e37d5367d4e800bdadf064d3bbde13e02b40b0ce41d1ca2 not found: ID does not exist" containerID="d92bf2f2e03a81889e37d5367d4e800bdadf064d3bbde13e02b40b0ce41d1ca2" Jul 31 15:17:51.794235 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.794199 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92bf2f2e03a81889e37d5367d4e800bdadf064d3bbde13e02b40b0ce41d1ca2"} err="failed to get container status \"d92bf2f2e03a81889e37d5367d4e800bdadf064d3bbde13e02b40b0ce41d1ca2\": rpc error: code = NotFound desc = could not find container \"d92bf2f2e03a81889e37d5367d4e800bdadf064d3bbde13e02b40b0ce41d1ca2\": container with ID starting with d92bf2f2e03a81889e37d5367d4e800bdadf064d3bbde13e02b40b0ce41d1ca2 not found: ID does not exist" Jul 31 15:17:51.794235 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.794218 2577 scope.go:117] "RemoveContainer" containerID="0a4c6fc43b63230a2d770aa53e664e5f99aa4c7b17a9bec405243e53ebaa9a0a" Jul 31 15:17:51.794437 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:17:51.794422 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a4c6fc43b63230a2d770aa53e664e5f99aa4c7b17a9bec405243e53ebaa9a0a\": container with ID starting with 0a4c6fc43b63230a2d770aa53e664e5f99aa4c7b17a9bec405243e53ebaa9a0a not found: ID does not exist" containerID="0a4c6fc43b63230a2d770aa53e664e5f99aa4c7b17a9bec405243e53ebaa9a0a" Jul 31 15:17:51.794473 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.794440 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4c6fc43b63230a2d770aa53e664e5f99aa4c7b17a9bec405243e53ebaa9a0a"} err="failed to get container status \"0a4c6fc43b63230a2d770aa53e664e5f99aa4c7b17a9bec405243e53ebaa9a0a\": rpc error: code = NotFound desc = could not find container \"0a4c6fc43b63230a2d770aa53e664e5f99aa4c7b17a9bec405243e53ebaa9a0a\": container with ID starting with 0a4c6fc43b63230a2d770aa53e664e5f99aa4c7b17a9bec405243e53ebaa9a0a not found: ID does not exist" Jul 31 15:17:51.798213 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.798190 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw"] Jul 31 15:17:51.802236 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:51.802215 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-9966b2-openshift-default-67478cf68d-jtrzw"] Jul 31 15:17:52.429478 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:52.429445 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a78760f-ff69-4f1c-800f-8c9cac414b13" path="/var/lib/kubelet/pods/8a78760f-ff69-4f1c-800f-8c9cac414b13/volumes" Jul 31 15:17:59.073917 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:17:59.073880 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bqzgx"] Jul 31 15:18:03.412357 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:03.412324 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv"] Jul 31 15:18:03.412738 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:03.412553 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" podUID="d81b6908-3ef9-407b-8f6b-28b7bc428311" containerName="istio-proxy" containerID="cri-o://bdb72d107987cb93db5ac7b4923d965a3c73d8fb5f049f180786d77357184f77" gracePeriod=30 Jul 31 15:18:08.663749 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.663722 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:18:08.781160 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.781057 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-credential-socket\") pod \"d81b6908-3ef9-407b-8f6b-28b7bc428311\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " Jul 31 15:18:08.781160 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.781129 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81b6908-3ef9-407b-8f6b-28b7bc428311-istiod-ca-cert\") pod \"d81b6908-3ef9-407b-8f6b-28b7bc428311\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " Jul 31 15:18:08.781399 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.781164 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-envoy\") pod \"d81b6908-3ef9-407b-8f6b-28b7bc428311\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " Jul 31 15:18:08.781399 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.781204 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-token\") pod \"d81b6908-3ef9-407b-8f6b-28b7bc428311\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " Jul 31 15:18:08.781399 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.781231 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qd4w\" (UniqueName: \"kubernetes.io/projected/d81b6908-3ef9-407b-8f6b-28b7bc428311-kube-api-access-8qd4w\") pod \"d81b6908-3ef9-407b-8f6b-28b7bc428311\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " Jul 31 15:18:08.781399 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.781269 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-workload-certs\") pod \"d81b6908-3ef9-407b-8f6b-28b7bc428311\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " Jul 31 15:18:08.781399 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.781299 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-workload-socket\") pod \"d81b6908-3ef9-407b-8f6b-28b7bc428311\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " Jul 31 15:18:08.781399 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.781348 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-data\") pod \"d81b6908-3ef9-407b-8f6b-28b7bc428311\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " Jul 31 15:18:08.781399 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.781351 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d81b6908-3ef9-407b-8f6b-28b7bc428311" (UID: "d81b6908-3ef9-407b-8f6b-28b7bc428311"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:18:08.781399 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.781387 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-podinfo\") pod \"d81b6908-3ef9-407b-8f6b-28b7bc428311\" (UID: \"d81b6908-3ef9-407b-8f6b-28b7bc428311\") " Jul 31 15:18:08.781806 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.781533 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d81b6908-3ef9-407b-8f6b-28b7bc428311" (UID: "d81b6908-3ef9-407b-8f6b-28b7bc428311"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:18:08.781806 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.781586 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d81b6908-3ef9-407b-8f6b-28b7bc428311" (UID: "d81b6908-3ef9-407b-8f6b-28b7bc428311"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:18:08.781806 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.781600 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-data" (OuterVolumeSpecName: "istio-data") pod "d81b6908-3ef9-407b-8f6b-28b7bc428311" (UID: "d81b6908-3ef9-407b-8f6b-28b7bc428311"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:18:08.781806 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.781646 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81b6908-3ef9-407b-8f6b-28b7bc428311-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d81b6908-3ef9-407b-8f6b-28b7bc428311" (UID: "d81b6908-3ef9-407b-8f6b-28b7bc428311"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:18:08.781806 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.781709 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-workload-certs\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:18:08.781806 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.781728 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-workload-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:18:08.781806 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.781742 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-data\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:18:08.781806 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.781756 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-credential-socket\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:18:08.783464 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.783429 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-token" (OuterVolumeSpecName: "istio-token") pod "d81b6908-3ef9-407b-8f6b-28b7bc428311" (UID: "d81b6908-3ef9-407b-8f6b-28b7bc428311"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:18:08.783585 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.783505 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81b6908-3ef9-407b-8f6b-28b7bc428311-kube-api-access-8qd4w" (OuterVolumeSpecName: "kube-api-access-8qd4w") pod "d81b6908-3ef9-407b-8f6b-28b7bc428311" (UID: "d81b6908-3ef9-407b-8f6b-28b7bc428311"). InnerVolumeSpecName "kube-api-access-8qd4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:18:08.783794 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.783769 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d81b6908-3ef9-407b-8f6b-28b7bc428311" (UID: "d81b6908-3ef9-407b-8f6b-28b7bc428311"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 15:18:08.783794 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.783766 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d81b6908-3ef9-407b-8f6b-28b7bc428311" (UID: "d81b6908-3ef9-407b-8f6b-28b7bc428311"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:18:08.858604 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.858567 2577 generic.go:356] "Generic (PLEG): container finished" podID="d81b6908-3ef9-407b-8f6b-28b7bc428311" containerID="bdb72d107987cb93db5ac7b4923d965a3c73d8fb5f049f180786d77357184f77" exitCode=0 Jul 31 15:18:08.858785 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.858619 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" Jul 31 15:18:08.858785 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.858662 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" event={"ID":"d81b6908-3ef9-407b-8f6b-28b7bc428311","Type":"ContainerDied","Data":"bdb72d107987cb93db5ac7b4923d965a3c73d8fb5f049f180786d77357184f77"} Jul 31 15:18:08.858785 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.858718 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv" event={"ID":"d81b6908-3ef9-407b-8f6b-28b7bc428311","Type":"ContainerDied","Data":"3f33a61a3cd95087f3929acbd9929bbd883a64709564b740d4f5409970bbe9a7"} Jul 31 15:18:08.858785 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.858746 2577 scope.go:117] "RemoveContainer" containerID="bdb72d107987cb93db5ac7b4923d965a3c73d8fb5f049f180786d77357184f77" Jul 31 15:18:08.868059 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.868042 2577 scope.go:117] "RemoveContainer" containerID="052e11ed3c4ba642cb17826ad2502ac23bea8256628c4607e3ac529c86f9bf67" Jul 31 15:18:08.875903 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.875888 2577 scope.go:117] "RemoveContainer" containerID="bdb72d107987cb93db5ac7b4923d965a3c73d8fb5f049f180786d77357184f77" Jul 31 15:18:08.876171 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:18:08.876149 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb72d107987cb93db5ac7b4923d965a3c73d8fb5f049f180786d77357184f77\": container with ID starting with bdb72d107987cb93db5ac7b4923d965a3c73d8fb5f049f180786d77357184f77 not found: ID does not exist" containerID="bdb72d107987cb93db5ac7b4923d965a3c73d8fb5f049f180786d77357184f77" Jul 31 15:18:08.876271 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.876176 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb72d107987cb93db5ac7b4923d965a3c73d8fb5f049f180786d77357184f77"} err="failed to get container status \"bdb72d107987cb93db5ac7b4923d965a3c73d8fb5f049f180786d77357184f77\": rpc error: code = NotFound desc = could not find container \"bdb72d107987cb93db5ac7b4923d965a3c73d8fb5f049f180786d77357184f77\": container with ID starting with bdb72d107987cb93db5ac7b4923d965a3c73d8fb5f049f180786d77357184f77 not found: ID does not exist" Jul 31 15:18:08.876271 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.876193 2577 scope.go:117] "RemoveContainer" containerID="052e11ed3c4ba642cb17826ad2502ac23bea8256628c4607e3ac529c86f9bf67" Jul 31 15:18:08.876404 ip-10-0-132-38 kubenswrapper[2577]: E0731 15:18:08.876389 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052e11ed3c4ba642cb17826ad2502ac23bea8256628c4607e3ac529c86f9bf67\": container with ID starting with 052e11ed3c4ba642cb17826ad2502ac23bea8256628c4607e3ac529c86f9bf67 not found: ID does not exist" containerID="052e11ed3c4ba642cb17826ad2502ac23bea8256628c4607e3ac529c86f9bf67" Jul 31 15:18:08.876445 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.876407 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052e11ed3c4ba642cb17826ad2502ac23bea8256628c4607e3ac529c86f9bf67"} err="failed to get container status \"052e11ed3c4ba642cb17826ad2502ac23bea8256628c4607e3ac529c86f9bf67\": rpc error: code = NotFound desc = could not find container \"052e11ed3c4ba642cb17826ad2502ac23bea8256628c4607e3ac529c86f9bf67\": container with ID starting with 052e11ed3c4ba642cb17826ad2502ac23bea8256628c4607e3ac529c86f9bf67 not found: ID does not exist" Jul 31 15:18:08.881939 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.881915 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv"] Jul 31 15:18:08.882462 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.882445 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-podinfo\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:18:08.882574 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.882555 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81b6908-3ef9-407b-8f6b-28b7bc428311-istiod-ca-cert\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:18:08.882574 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.882575 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-envoy\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:18:08.882760 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.882586 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81b6908-3ef9-407b-8f6b-28b7bc428311-istio-token\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:18:08.882760 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.882597 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qd4w\" (UniqueName: \"kubernetes.io/projected/d81b6908-3ef9-407b-8f6b-28b7bc428311-kube-api-access-8qd4w\") on node \"ip-10-0-132-38.ec2.internal\" DevicePath \"\"" Jul 31 15:18:08.896448 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:08.896417 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-79a3ee-openshift-default-5b58b78cfb-bwtlv"] Jul 31 15:18:10.429990 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:10.429957 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d81b6908-3ef9-407b-8f6b-28b7bc428311" path="/var/lib/kubelet/pods/d81b6908-3ef9-407b-8f6b-28b7bc428311/volumes" Jul 31 15:18:10.479719 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:10.479693 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-698c94c45b-pkqb4_bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8/copy-manifests/0.log" Jul 31 15:18:10.487240 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:10.487222 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-698c94c45b-pkqb4_bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8/manager/0.log" Jul 31 15:18:11.080062 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:11.080035 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29758500-mbqrf_de570bb7-92e1-484c-9ef9-cd1ca5b4d6d8/cleanup/0.log" Jul 31 15:18:12.711662 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:12.711629 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-bqzgx_021631ac-a079-4b7d-bf1d-42bc81796fc0/limitador/0.log" Jul 31 15:18:13.320701 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:13.320667 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6k8v4_431d2c6a-3bd3-4d06-af77-417f21012958/discovery/0.log" Jul 31 15:18:14.509321 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:14.509291 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-xkd9f_3880ffab-f747-47fb-bfac-d7eb278eb474/main/0.log" Jul 31 15:18:23.668908 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:23.668820 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lsrr4_72dad2d5-e7ca-4509-8619-e30eb609e348/global-pull-secret-syncer/0.log" Jul 31 15:18:23.833385 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:23.833355 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hclhf_6a540518-e608-4021-a467-eaa910669373/konnectivity-agent/0.log" Jul 31 15:18:23.925331 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:23.925254 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-38.ec2.internal_244611c7bbcc8364a26d08e8f81e9ee4/haproxy/0.log" Jul 31 15:18:29.006595 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:29.006564 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-bqzgx_021631ac-a079-4b7d-bf1d-42bc81796fc0/limitador/0.log" Jul 31 15:18:30.828885 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:30.828856 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6b49c5d89f-kbzr2_3d3df8bb-e9bc-42ff-af0c-efeef3d9b0c4/metrics-server/0.log" Jul 31 15:18:31.061492 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:31.061410 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l2kdv_9f0c43be-78de-4de3-99a1-e1001f6208c8/node-exporter/0.log" Jul 31 15:18:31.083412 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:31.083384 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l2kdv_9f0c43be-78de-4de3-99a1-e1001f6208c8/kube-rbac-proxy/0.log" Jul 31 15:18:31.108002 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:31.107976 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l2kdv_9f0c43be-78de-4de3-99a1-e1001f6208c8/init-textfile/0.log" Jul 31 15:18:31.138584 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:31.138559 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-wlzs6_ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd/kube-rbac-proxy-main/0.log" Jul 31 15:18:31.171308 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:31.171282 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-wlzs6_ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd/kube-rbac-proxy-self/0.log" Jul 31 15:18:31.191536 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:31.191506 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-wlzs6_ec8ae95a-3a6e-476a-aafb-78b54aa6ddcd/openshift-state-metrics/0.log" Jul 31 15:18:31.229741 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:31.229717 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8cc199f-3ca5-45c4-8e3f-7a4768f80e24/prometheus/0.log" Jul 31 15:18:31.251195 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:31.251164 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8cc199f-3ca5-45c4-8e3f-7a4768f80e24/config-reloader/0.log" Jul 31 15:18:31.275617 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:31.275591 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8cc199f-3ca5-45c4-8e3f-7a4768f80e24/thanos-sidecar/0.log" Jul 31 15:18:31.309659 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:31.309634 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8cc199f-3ca5-45c4-8e3f-7a4768f80e24/kube-rbac-proxy-web/0.log" Jul 31 15:18:31.339261 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:31.339191 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8cc199f-3ca5-45c4-8e3f-7a4768f80e24/kube-rbac-proxy/0.log" Jul 31 15:18:31.358863 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:31.358822 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8cc199f-3ca5-45c4-8e3f-7a4768f80e24/kube-rbac-proxy-thanos/0.log" Jul 31 15:18:31.380399 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:31.380378 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8cc199f-3ca5-45c4-8e3f-7a4768f80e24/init-config-reloader/0.log" Jul 31 15:18:34.246439 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246400 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc"] Jul 31 15:18:34.246954 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246794 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4feaae1a-54a5-4fc7-843e-d0c82255c13a" containerName="payload-pre-processing" Jul 31 15:18:34.246954 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246811 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4feaae1a-54a5-4fc7-843e-d0c82255c13a" containerName="payload-pre-processing" Jul 31 15:18:34.246954 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246827 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10dd78e5-a052-45f1-805c-8f7a9a1db0fa" containerName="payload-pre-processing" Jul 31 15:18:34.246954 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246835 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="10dd78e5-a052-45f1-805c-8f7a9a1db0fa" containerName="payload-pre-processing" Jul 31 15:18:34.246954 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246866 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99add764-c1dd-428a-bb04-a6f4ed6599cf" containerName="payload-processing" Jul 31 15:18:34.246954 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246876 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99add764-c1dd-428a-bb04-a6f4ed6599cf" containerName="payload-processing" Jul 31 15:18:34.246954 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246888 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de459d3d-e0b4-4810-a31b-666efc74cae2" containerName="istio-proxy" Jul 31 15:18:34.246954 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246896 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="de459d3d-e0b4-4810-a31b-666efc74cae2" containerName="istio-proxy" Jul 31 15:18:34.246954 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246906 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b4a38ac-d1ec-439e-aa0d-c498508794ee" containerName="payload-processing" Jul 31 15:18:34.246954 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246913 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b4a38ac-d1ec-439e-aa0d-c498508794ee" containerName="payload-processing" Jul 31 15:18:34.246954 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246923 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a78760f-ff69-4f1c-800f-8c9cac414b13" containerName="istio-proxy" Jul 31 15:18:34.246954 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246931 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a78760f-ff69-4f1c-800f-8c9cac414b13" containerName="istio-proxy" Jul 31 15:18:34.246954 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246938 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81b6908-3ef9-407b-8f6b-28b7bc428311" containerName="istio-proxy" Jul 31 15:18:34.246954 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246945 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81b6908-3ef9-407b-8f6b-28b7bc428311" containerName="istio-proxy" Jul 31 15:18:34.246954 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246955 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de459d3d-e0b4-4810-a31b-666efc74cae2" containerName="istio-proxy" Jul 31 15:18:34.246954 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246962 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="de459d3d-e0b4-4810-a31b-666efc74cae2" containerName="istio-proxy" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246983 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f" containerName="payload-processing" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.246991 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f" containerName="payload-processing" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.247004 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06e3de25-5249-4bfd-bf75-a0c0b013b660" containerName="payload-pre-processing" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.247014 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="06e3de25-5249-4bfd-bf75-a0c0b013b660" containerName="payload-pre-processing" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.247030 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81b6908-3ef9-407b-8f6b-28b7bc428311" containerName="istio-proxy" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.247039 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81b6908-3ef9-407b-8f6b-28b7bc428311" containerName="istio-proxy" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.247054 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a78760f-ff69-4f1c-800f-8c9cac414b13" containerName="istio-proxy" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.247063 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a78760f-ff69-4f1c-800f-8c9cac414b13" containerName="istio-proxy" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.247166 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81b6908-3ef9-407b-8f6b-28b7bc428311" containerName="istio-proxy" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.247179 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89b4ba06-83bd-4a7f-b3a6-1d910d3ef42f" containerName="payload-processing" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.247188 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="de459d3d-e0b4-4810-a31b-666efc74cae2" containerName="istio-proxy" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.247197 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="06e3de25-5249-4bfd-bf75-a0c0b013b660" containerName="payload-pre-processing" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.247206 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="10dd78e5-a052-45f1-805c-8f7a9a1db0fa" containerName="payload-pre-processing" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.247215 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="99add764-c1dd-428a-bb04-a6f4ed6599cf" containerName="payload-processing" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.247225 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a78760f-ff69-4f1c-800f-8c9cac414b13" containerName="istio-proxy" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.247233 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b4a38ac-d1ec-439e-aa0d-c498508794ee" containerName="payload-processing" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.247242 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a78760f-ff69-4f1c-800f-8c9cac414b13" containerName="istio-proxy" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.247253 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4feaae1a-54a5-4fc7-843e-d0c82255c13a" containerName="payload-pre-processing" Jul 31 15:18:34.247727 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.247261 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="de459d3d-e0b4-4810-a31b-666efc74cae2" containerName="istio-proxy" Jul 31 15:18:34.251924 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.251905 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:34.254213 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.254195 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l7w47\"/\"kube-root-ca.crt\"" Jul 31 15:18:34.254595 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.254579 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l7w47\"/\"default-dockercfg-8djw8\"" Jul 31 15:18:34.254645 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.254606 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l7w47\"/\"openshift-service-ca.crt\"" Jul 31 15:18:34.262045 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.262021 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc"] Jul 31 15:18:34.287944 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.287914 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbttg\" (UniqueName: \"kubernetes.io/projected/f3dc12c7-18b4-47f1-a546-3574ccee3d39-kube-api-access-kbttg\") pod \"perf-node-gather-daemonset-5dsnc\" (UID: \"f3dc12c7-18b4-47f1-a546-3574ccee3d39\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:34.288133 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.287953 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3dc12c7-18b4-47f1-a546-3574ccee3d39-proc\") pod \"perf-node-gather-daemonset-5dsnc\" (UID: \"f3dc12c7-18b4-47f1-a546-3574ccee3d39\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:34.288133 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.287971 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3dc12c7-18b4-47f1-a546-3574ccee3d39-sys\") pod \"perf-node-gather-daemonset-5dsnc\" (UID: \"f3dc12c7-18b4-47f1-a546-3574ccee3d39\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:34.288133 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.287997 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3dc12c7-18b4-47f1-a546-3574ccee3d39-podres\") pod \"perf-node-gather-daemonset-5dsnc\" (UID: \"f3dc12c7-18b4-47f1-a546-3574ccee3d39\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:34.288133 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.288108 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3dc12c7-18b4-47f1-a546-3574ccee3d39-lib-modules\") pod \"perf-node-gather-daemonset-5dsnc\" (UID: \"f3dc12c7-18b4-47f1-a546-3574ccee3d39\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:34.388855 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.388812 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3dc12c7-18b4-47f1-a546-3574ccee3d39-lib-modules\") pod \"perf-node-gather-daemonset-5dsnc\" (UID: \"f3dc12c7-18b4-47f1-a546-3574ccee3d39\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:34.389055 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.388877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbttg\" (UniqueName: \"kubernetes.io/projected/f3dc12c7-18b4-47f1-a546-3574ccee3d39-kube-api-access-kbttg\") pod \"perf-node-gather-daemonset-5dsnc\" (UID: \"f3dc12c7-18b4-47f1-a546-3574ccee3d39\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:34.389055 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.388903 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3dc12c7-18b4-47f1-a546-3574ccee3d39-proc\") pod \"perf-node-gather-daemonset-5dsnc\" (UID: \"f3dc12c7-18b4-47f1-a546-3574ccee3d39\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:34.389055 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.388921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3dc12c7-18b4-47f1-a546-3574ccee3d39-sys\") pod \"perf-node-gather-daemonset-5dsnc\" (UID: \"f3dc12c7-18b4-47f1-a546-3574ccee3d39\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:34.389055 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.388942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3dc12c7-18b4-47f1-a546-3574ccee3d39-podres\") pod \"perf-node-gather-daemonset-5dsnc\" (UID: \"f3dc12c7-18b4-47f1-a546-3574ccee3d39\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:34.389055 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.389007 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3dc12c7-18b4-47f1-a546-3574ccee3d39-proc\") pod \"perf-node-gather-daemonset-5dsnc\" (UID: \"f3dc12c7-18b4-47f1-a546-3574ccee3d39\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:34.389055 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.389008 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3dc12c7-18b4-47f1-a546-3574ccee3d39-lib-modules\") pod \"perf-node-gather-daemonset-5dsnc\" (UID: \"f3dc12c7-18b4-47f1-a546-3574ccee3d39\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:34.389055 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.389036 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3dc12c7-18b4-47f1-a546-3574ccee3d39-sys\") pod \"perf-node-gather-daemonset-5dsnc\" (UID: \"f3dc12c7-18b4-47f1-a546-3574ccee3d39\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:34.389387 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.389064 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3dc12c7-18b4-47f1-a546-3574ccee3d39-podres\") pod \"perf-node-gather-daemonset-5dsnc\" (UID: \"f3dc12c7-18b4-47f1-a546-3574ccee3d39\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:34.398731 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.398708 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbttg\" (UniqueName: \"kubernetes.io/projected/f3dc12c7-18b4-47f1-a546-3574ccee3d39-kube-api-access-kbttg\") pod \"perf-node-gather-daemonset-5dsnc\" (UID: \"f3dc12c7-18b4-47f1-a546-3574ccee3d39\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:34.561974 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.561866 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:34.687158 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.687059 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc"] Jul 31 15:18:34.689898 ip-10-0-132-38 kubenswrapper[2577]: W0731 15:18:34.689867 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf3dc12c7_18b4_47f1_a546_3574ccee3d39.slice/crio-c569ef9399a786974fc40c5b71d1bccabb0dea050b0a36df96764b6e4de03d02 WatchSource:0}: Error finding container c569ef9399a786974fc40c5b71d1bccabb0dea050b0a36df96764b6e4de03d02: Status 404 returned error can't find the container with id c569ef9399a786974fc40c5b71d1bccabb0dea050b0a36df96764b6e4de03d02 Jul 31 15:18:34.971722 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.971687 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" event={"ID":"f3dc12c7-18b4-47f1-a546-3574ccee3d39","Type":"ContainerStarted","Data":"58fac45ac857343ccf1821cde4561e1a193f5328204dc3af0fe61c01ef208cdd"} Jul 31 15:18:34.971927 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.971738 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" event={"ID":"f3dc12c7-18b4-47f1-a546-3574ccee3d39","Type":"ContainerStarted","Data":"c569ef9399a786974fc40c5b71d1bccabb0dea050b0a36df96764b6e4de03d02"} Jul 31 15:18:34.971927 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.971776 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:34.986070 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:34.985978 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" podStartSLOduration=0.985956758 podStartE2EDuration="985.956758ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 15:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:18:34.985563877 +0000 UTC m=+3967.022386944" watchObservedRunningTime="2026-07-31 15:18:34.985956758 +0000 UTC m=+3967.022779825" Jul 31 15:18:36.148466 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:36.148433 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6lm6j_7655bcd6-8ab3-4f31-bd58-7ebfef6f840b/dns/0.log" Jul 31 15:18:36.167675 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:36.167653 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6lm6j_7655bcd6-8ab3-4f31-bd58-7ebfef6f840b/kube-rbac-proxy/0.log" Jul 31 15:18:36.291891 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:36.291860 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f2v62_80dc0e3d-60fa-46e3-9e00-20eefb9986eb/dns-node-resolver/0.log" Jul 31 15:18:36.948305 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:36.948273 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qltd5_0fd2b5b1-f721-4a2e-8c19-1c220634d162/node-ca/0.log" Jul 31 15:18:38.064666 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:38.064635 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6k8v4_431d2c6a-3bd3-4d06-af77-417f21012958/discovery/0.log" Jul 31 15:18:39.067356 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:39.067322 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8l2tx_43e35457-db4f-45d4-a5ac-dd16537435d6/serve-healthcheck-canary/0.log" Jul 31 15:18:39.659792 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:39.659762 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fr5dq_7143a362-9b64-4ff5-aba1-40effb7058c0/kube-rbac-proxy/0.log" Jul 31 15:18:39.678637 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:39.678602 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fr5dq_7143a362-9b64-4ff5-aba1-40effb7058c0/exporter/0.log" Jul 31 15:18:39.697688 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:39.697658 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fr5dq_7143a362-9b64-4ff5-aba1-40effb7058c0/extractor/0.log" Jul 31 15:18:40.986453 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:40.986423 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-5dsnc" Jul 31 15:18:42.870470 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:42.870438 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-698c94c45b-pkqb4_bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8/manager/0.log" Jul 31 15:18:42.900878 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:42.900834 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-698c94c45b-pkqb4_bb0d24d0-5cb8-4ee0-9e98-949dfabc8ec8/copy-manifests/0.log" Jul 31 15:18:44.353836 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:44.353807 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-58f6dff6d5-ckp22_6a392d13-3092-4a3c-abd2-b84be867aa00/manager/0.log" Jul 31 15:18:44.957693 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:44.957659 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6k8v4_431d2c6a-3bd3-4d06-af77-417f21012958/discovery/0.log" Jul 31 15:18:46.736146 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:46.736117 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6k8v4_431d2c6a-3bd3-4d06-af77-417f21012958/discovery/0.log" Jul 31 15:18:50.973668 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:50.973639 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g9plr_1a3824fc-d397-431c-8796-c494dc5d022b/kube-multus-additional-cni-plugins/0.log" Jul 31 15:18:50.992206 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:50.992179 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g9plr_1a3824fc-d397-431c-8796-c494dc5d022b/egress-router-binary-copy/0.log" Jul 31 15:18:51.010814 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:51.010790 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g9plr_1a3824fc-d397-431c-8796-c494dc5d022b/cni-plugins/0.log" Jul 31 15:18:51.030459 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:51.030435 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g9plr_1a3824fc-d397-431c-8796-c494dc5d022b/bond-cni-plugin/0.log" Jul 31 15:18:51.048597 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:51.048573 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g9plr_1a3824fc-d397-431c-8796-c494dc5d022b/routeoverride-cni/0.log" Jul 31 15:18:51.066856 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:51.066812 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g9plr_1a3824fc-d397-431c-8796-c494dc5d022b/whereabouts-cni-bincopy/0.log" Jul 31 15:18:51.084308 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:51.084274 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g9plr_1a3824fc-d397-431c-8796-c494dc5d022b/whereabouts-cni/0.log" Jul 31 15:18:51.316820 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:51.316745 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wx97w_c3047cea-d168-4307-a588-6c1f60e9a921/kube-multus/0.log" Jul 31 15:18:51.417971 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:51.417943 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-skd9g_78e66929-08a3-4aa1-8a96-740f8f8eb7c8/network-metrics-daemon/0.log" Jul 31 15:18:51.434079 ip-10-0-132-38 kubenswrapper[2577]: I0731 15:18:51.434055 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-skd9g_78e66929-08a3-4aa1-8a96-740f8f8eb7c8/kube-rbac-proxy/0.log"