Jul 20 17:46:01.020169 ip-10-0-141-129 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 20 17:46:01.020178 ip-10-0-141-129 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 20 17:46:01.020185 ip-10-0-141-129 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 20 17:46:01.020400 ip-10-0-141-129 systemd[1]: Failed to start Kubernetes Kubelet. Jul 20 17:46:11.270022 ip-10-0-141-129 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 20 17:46:11.270043 ip-10-0-141-129 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 9437e430da8f439ca8cabb3694611d95 -- Jul 20 17:48:11.519053 ip-10-0-141-129 systemd[1]: Starting Kubernetes Kubelet... Jul 20 17:48:11.893806 ip-10-0-141-129 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 17:48:11.893806 ip-10-0-141-129 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 20 17:48:11.893806 ip-10-0-141-129 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 17:48:11.893806 ip-10-0-141-129 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 20 17:48:11.893806 ip-10-0-141-129 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 17:48:11.894831 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.894683 2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 20 17:48:11.901400 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901384 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 17:48:11.901400 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901399 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901405 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901408 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901411 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901414 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901417 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901427 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901430 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901433 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901436 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901439 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901441 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901444 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901447 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901449 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901452 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901454 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901457 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901459 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901462 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 17:48:11.901462 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901464 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901467 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901470 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901473 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901475 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901478 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901481 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901483 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901486 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901488 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901491 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901493 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901496 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901498 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901501 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901503 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901505 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901508 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901511 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901513 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 17:48:11.901932 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901516 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901518 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901521 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901523 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901525 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901528 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901530 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901533 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901537 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901540 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901543 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901545 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901548 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901550 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901555 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901558 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901560 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901563 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901566 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901568 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 17:48:11.902459 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901571 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901573 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901576 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901578 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901581 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901583 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901585 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901589 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901592 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901596 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901599 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901602 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901605 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901607 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901610 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901612 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901615 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901617 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901620 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901622 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 17:48:11.902937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901625 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901627 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901631 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901633 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901636 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901638 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901642 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901645 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901647 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901650 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901653 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901656 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901658 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901661 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901665 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901668 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901671 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.901674 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902398 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 17:48:11.903404 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902405 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902409 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902413 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902415 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902418 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902421 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902424 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902426 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902429 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902431 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902434 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902436 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902439 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902441 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902444 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902447 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902449 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902452 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902454 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 17:48:11.903933 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902458 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902461 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902464 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902467 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902469 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902472 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902475 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902479 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902482 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902485 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902488 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902491 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902494 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902497 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902500 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902503 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902506 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902509 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902511 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 17:48:11.904607 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902515 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902518 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902520 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902523 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902525 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902528 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902530 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902533 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902535 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902537 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902540 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902542 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902545 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902548 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902551 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902554 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902557 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902559 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902562 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902564 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 17:48:11.905267 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902566 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902569 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902573 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902575 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902577 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902580 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902582 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902585 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902587 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902590 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902592 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902595 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902597 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902600 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902603 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902605 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902609 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902613 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902616 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 17:48:11.905771 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902619 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902621 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902625 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902628 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902630 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902633 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902636 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902639 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902642 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902644 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902646 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902650 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902652 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902655 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902657 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902660 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902663 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902665 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902668 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902670 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 17:48:11.906219 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.902673 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903101 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903108 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903112 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903115 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903117 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903121 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903125 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903127 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903130 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903133 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903135 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903138 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903141 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903143 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903145 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903148 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903151 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903154 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903157 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 17:48:11.906720 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903160 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903163 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903166 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903168 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903171 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903174 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903176 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903179 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903181 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903184 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903186 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903189 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903191 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903194 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903196 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903199 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903201 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903204 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903206 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903209 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 17:48:11.907226 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903212 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903215 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903217 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903219 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903222 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903224 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903227 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903229 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903232 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903234 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903237 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903239 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903242 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903246 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903248 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903252 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903254 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903257 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903259 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903262 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 17:48:11.907717 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903264 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903267 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903269 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903271 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903274 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903277 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903279 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903281 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903284 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903286 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903289 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903291 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903295 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903298 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903300 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903304 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903308 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903311 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903314 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 17:48:11.908218 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903317 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903319 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903322 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903325 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903327 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903330 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903332 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903335 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903337 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903339 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903342 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903345 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903347 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903349 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903352 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903356 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903360 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903362 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903365 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 17:48:11.908669 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903368 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.903371 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903439 2583 flags.go:64] FLAG: --address="0.0.0.0" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903454 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903464 2583 flags.go:64] FLAG: --anonymous-auth="true" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903471 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903475 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903479 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903484 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903488 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903492 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903495 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903498 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903502 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903505 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903508 2583 flags.go:64] FLAG: --cgroup-root="" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903511 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903514 2583 flags.go:64] FLAG: --client-ca-file="" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903516 2583 flags.go:64] FLAG: --cloud-provider="external" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903519 2583 flags.go:64] FLAG: --cluster-dns="[]" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903524 2583 flags.go:64] FLAG: --cluster-domain="" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903527 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903530 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903533 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 20 17:48:11.909131 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903536 2583 flags.go:64] FLAG: --container-log-max-files="5" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903540 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903543 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903546 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903549 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903552 2583 flags.go:64] FLAG: --contention-profiling="false" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903555 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903559 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903562 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903564 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903569 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903572 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903575 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903580 2583 flags.go:64] FLAG: --enable-load-reader="false" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903583 2583 flags.go:64] FLAG: --enable-server="true" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903586 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903591 2583 flags.go:64] FLAG: --event-burst="100" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903594 2583 flags.go:64] FLAG: --event-qps="50" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903597 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903600 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903603 2583 flags.go:64] FLAG: --eviction-hard="" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903606 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903609 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903612 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903615 2583 flags.go:64] FLAG: --eviction-soft="" Jul 20 17:48:11.909692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903618 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903621 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903624 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903627 2583 flags.go:64] FLAG: --experimental-mounter-path="" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903630 2583 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903633 2583 flags.go:64] FLAG: --fail-swap-on="true" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903636 2583 flags.go:64] FLAG: --feature-gates="" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903639 2583 flags.go:64] FLAG: --file-check-frequency="20s" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903642 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903645 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903648 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903651 2583 flags.go:64] FLAG: --healthz-port="10248" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903654 2583 flags.go:64] FLAG: --help="false" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903657 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-141-129.ec2.internal" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903660 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903663 2583 flags.go:64] FLAG: --http-check-frequency="20s" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903666 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903671 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903674 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903677 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903681 2583 flags.go:64] FLAG: --image-service-endpoint="" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903684 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903687 2583 flags.go:64] FLAG: --kube-api-burst="100" Jul 20 17:48:11.910334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903700 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903707 2583 flags.go:64] FLAG: --kube-api-qps="50" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903710 2583 flags.go:64] FLAG: --kube-reserved="" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903713 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903716 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903719 2583 flags.go:64] FLAG: --kubelet-cgroups="" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903722 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903740 2583 flags.go:64] FLAG: --lock-file="" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903743 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903746 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903749 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903755 2583 flags.go:64] FLAG: --log-json-split-stream="false" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903757 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903760 2583 flags.go:64] FLAG: --log-text-split-stream="false" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903763 2583 flags.go:64] FLAG: --logging-format="text" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903767 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903770 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903773 2583 flags.go:64] FLAG: --manifest-url="" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903776 2583 flags.go:64] FLAG: --manifest-url-header="" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903782 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903785 2583 flags.go:64] FLAG: --max-open-files="1000000" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903789 2583 flags.go:64] FLAG: --max-pods="110" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903792 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903795 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903798 2583 flags.go:64] FLAG: --memory-manager-policy="None" Jul 20 17:48:11.910925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903801 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903806 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903809 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903812 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903819 2583 flags.go:64] FLAG: --node-status-max-images="50" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903822 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903826 2583 flags.go:64] FLAG: --oom-score-adj="-999" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903829 2583 flags.go:64] FLAG: --pod-cidr="" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903831 2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903838 2583 flags.go:64] FLAG: --pod-manifest-path="" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903840 2583 flags.go:64] FLAG: --pod-max-pids="-1" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903843 2583 flags.go:64] FLAG: --pods-per-core="0" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903846 2583 flags.go:64] FLAG: --port="10250" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903849 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903852 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b6be60d930e326bd" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903855 2583 flags.go:64] FLAG: --qos-reserved="" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903858 2583 flags.go:64] FLAG: --read-only-port="10255" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903861 2583 flags.go:64] FLAG: --register-node="true" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903864 2583 flags.go:64] FLAG: --register-with-taints="" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903867 2583 flags.go:64] FLAG: --registry-burst="10" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903870 2583 flags.go:64] FLAG: --registry-qps="5" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903873 2583 flags.go:64] FLAG: --reserved-cpus="" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903937 2583 flags.go:64] FLAG: --reserved-memory="" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903947 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903954 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 20 17:48:11.911641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903960 2583 flags.go:64] FLAG: --rotate-certificates="false" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903965 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903971 2583 flags.go:64] FLAG: --runonce="false" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903975 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903981 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903992 2583 flags.go:64] FLAG: --seccomp-default="false" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.903998 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.904003 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.904010 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.904015 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.904020 2583 flags.go:64] FLAG: --storage-driver-password="root" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.904024 2583 flags.go:64] FLAG: --storage-driver-secure="false" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905439 2583 flags.go:64] FLAG: --storage-driver-table="stats" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905449 2583 flags.go:64] FLAG: --storage-driver-user="root" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905457 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905509 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905643 2583 flags.go:64] FLAG: --system-cgroups="" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905649 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905660 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905663 2583 flags.go:64] FLAG: --tls-cert-file="" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905667 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905672 2583 flags.go:64] FLAG: --tls-min-version="" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905676 2583 flags.go:64] FLAG: --tls-private-key-file="" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905679 2583 flags.go:64] FLAG: --topology-manager-policy="none" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905682 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 20 17:48:11.912272 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905685 2583 flags.go:64] FLAG: --topology-manager-scope="container" Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905688 2583 flags.go:64] FLAG: --v="2" Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905703 2583 flags.go:64] FLAG: --version="false" Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905707 2583 flags.go:64] FLAG: --vmodule="" Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905710 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.905714 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905845 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905850 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905853 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905857 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905862 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905865 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905868 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905871 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905874 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905877 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905880 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905883 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905886 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905889 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905891 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 17:48:11.912908 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905894 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905897 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905901 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905904 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905906 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905909 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905912 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905914 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905917 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905920 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905922 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905925 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905927 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905930 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905933 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905935 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905938 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905940 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905943 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905946 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 17:48:11.913409 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905948 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905950 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905953 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905955 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905958 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905960 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905963 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905965 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905968 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905970 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905973 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905976 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905978 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905981 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905984 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905987 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905990 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905992 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905995 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.905997 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 17:48:11.913944 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906000 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906003 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906005 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906008 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906010 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906013 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906015 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906018 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906021 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906024 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906026 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906029 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906031 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906034 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906036 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906039 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906041 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906044 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906046 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906049 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906051 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 17:48:11.914443 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906053 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 17:48:11.914965 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906056 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 17:48:11.914965 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906059 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 17:48:11.914965 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906062 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 17:48:11.914965 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906065 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 17:48:11.914965 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906068 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 17:48:11.914965 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906072 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 17:48:11.914965 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906077 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 17:48:11.914965 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906080 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 17:48:11.914965 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906083 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 17:48:11.914965 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906086 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 17:48:11.914965 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906089 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 17:48:11.914965 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906091 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 17:48:11.914965 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906095 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 17:48:11.914965 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906098 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 17:48:11.914965 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906101 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 17:48:11.914965 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906104 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 17:48:11.914965 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906107 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 17:48:11.914965 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906110 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 17:48:11.915397 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906113 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 17:48:11.915397 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906116 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 17:48:11.915397 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906119 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 17:48:11.915397 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.906121 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 17:48:11.915397 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.906127 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 17:48:11.915397 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.912737 2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 20 17:48:11.915397 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.912754 2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 20 17:48:11.915397 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.912773 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 20 17:48:11.915397 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.912778 2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 20 17:48:11.915397 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912832 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 17:48:11.915397 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912838 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 17:48:11.915397 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912842 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 17:48:11.915397 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912845 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 17:48:11.915397 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912848 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912851 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912854 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912857 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912859 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912862 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912867 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912870 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912873 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912875 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912878 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912881 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912884 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912886 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912889 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912891 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912894 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912897 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912900 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912902 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 17:48:11.915770 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912905 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912908 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912911 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912913 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912916 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912920 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912923 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912926 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912928 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912931 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912933 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912936 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912940 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912944 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912949 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912952 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912955 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912959 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912963 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 17:48:11.916264 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912966 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912969 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912972 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912974 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912977 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912980 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912982 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912985 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912987 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912990 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912992 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912995 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.912997 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913000 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913003 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913005 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913008 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913010 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913014 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913017 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 17:48:11.916722 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913019 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913022 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913025 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913028 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913030 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913033 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913036 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913038 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913041 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913043 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913046 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913048 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913051 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913054 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913056 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913059 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913061 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913064 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913067 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913069 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 17:48:11.917269 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913071 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 20 17:48:11.917749 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913074 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 17:48:11.917749 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913077 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 17:48:11.917749 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913079 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 17:48:11.917749 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913082 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 17:48:11.917749 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913085 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 17:48:11.917749 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913087 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 17:48:11.917749 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913090 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 17:48:11.917749 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913092 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 17:48:11.917749 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913095 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 17:48:11.917749 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913097 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 17:48:11.917749 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913100 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 17:48:11.917749 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913102 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 17:48:11.917749 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913105 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 17:48:11.917749 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913107 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 17:48:11.917749 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913110 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 17:48:11.918125 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.913115 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 17:48:11.918125 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913227 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 17:48:11.918125 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913232 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 17:48:11.918125 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913236 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 17:48:11.918125 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913239 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 17:48:11.918125 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913242 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 17:48:11.918125 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913245 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 17:48:11.918125 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913248 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 17:48:11.918125 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913250 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 17:48:11.918125 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913253 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 17:48:11.918125 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913257 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 17:48:11.918125 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913259 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 17:48:11.918125 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913262 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 17:48:11.918125 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913265 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 17:48:11.918125 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913268 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913270 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913273 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913275 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913278 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913280 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913283 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913285 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913288 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913290 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913292 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913295 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913298 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913300 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913302 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913305 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913307 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913310 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913312 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913315 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913317 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 17:48:11.918492 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913320 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 17:48:11.919018 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913322 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 17:48:11.919018 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913325 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 17:48:11.919018 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913327 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 17:48:11.919018 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913329 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 17:48:11.919018 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913332 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 17:48:11.919018 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913335 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 17:48:11.919018 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913337 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 17:48:11.919018 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913340 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 17:48:11.919018 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913343 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 17:48:11.919018 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913345 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 17:48:11.919018 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913348 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 17:48:11.919018 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913351 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 17:48:11.919018 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913354 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 17:48:11.919018 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913357 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 17:48:11.919018 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913360 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 17:48:11.919018 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913362 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 17:48:11.919018 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913365 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 17:48:11.919018 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913369 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913372 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913375 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913378 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913381 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913384 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913386 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913388 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913391 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913393 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913396 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913398 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913401 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913404 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913407 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913409 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913412 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913414 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913417 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913420 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 17:48:11.919449 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913423 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913426 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913428 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913436 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913438 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913441 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913443 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913446 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913448 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913451 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913453 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913456 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913459 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913461 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913463 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913466 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913468 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913471 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913474 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913476 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 17:48:11.919998 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913478 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 17:48:11.920478 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913481 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 17:48:11.920478 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913483 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 17:48:11.920478 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913486 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 17:48:11.920478 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913488 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 17:48:11.920478 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913491 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 17:48:11.920478 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:11.913493 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 17:48:11.920478 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.913498 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 17:48:11.920478 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.913594 2583 server.go:962] "Client rotation is on, will bootstrap in background" Jul 20 17:48:11.920478 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.915510 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 20 17:48:11.920478 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.916651 2583 server.go:1019] "Starting client certificate rotation" Jul 20 17:48:11.920478 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.916757 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 17:48:11.920478 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.916790 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 17:48:11.920478 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.918994 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 17:48:11.920824 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.920100 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 17:48:11.929959 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.929943 2583 log.go:25] "Validated CRI v1 runtime API" Jul 20 17:48:11.932129 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.932111 2583 log.go:25] "Validated CRI v1 image API" Jul 20 17:48:11.934885 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.934870 2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 20 17:48:11.937142 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.937122 2583 fs.go:135] Filesystem UUIDs: map[14571c09-c293-4103-b252-f2acaf4b5567:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d7222ebf-af98-4775-ad89-fa646a8b0666:/dev/nvme0n1p3] Jul 20 17:48:11.937218 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.937141 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 20 17:48:11.943451 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.943338 2583 manager.go:216] Machine: {Timestamp:2026-07-20 17:48:11.941940135 +0000 UTC m=+0.313728701 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100096 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a2ad679a4b4051521e16522214450 SystemUUID:ec2a2ad6-79a4-b405-1521-e16522214450 BootID:9437e430-da8f-439c-a8ca-bb3694611d95 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8f:1b:22:0e:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8f:1b:22:0e:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:1b:e1:c3:8c:84 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 20 17:48:11.943451 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.943443 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 20 17:48:11.943589 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.943521 2583 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 20 17:48:11.946694 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.946669 2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 20 17:48:11.946835 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.946698 2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-129.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 20 17:48:11.947341 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.947332 2583 topology_manager.go:138] "Creating topology manager with none policy" Jul 20 17:48:11.947376 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.947343 2583 container_manager_linux.go:309] "Creating device plugin manager" Jul 20 17:48:11.947376 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.947350 2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 17:48:11.948080 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.948068 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 17:48:11.949268 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.949259 2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 20 17:48:11.952073 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.952062 2583 state_mem.go:36] "Initialized new in-memory state store" Jul 20 17:48:11.952192 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.952183 2583 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 20 17:48:11.952893 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.952883 2583 kubelet.go:486] "Attempting to sync node with API server" Jul 20 17:48:11.952932 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.952897 2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 20 17:48:11.952932 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.952910 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 20 17:48:11.952932 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.952919 2583 kubelet.go:398] "Adding apiserver pod source" Jul 20 17:48:11.952932 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.952928 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 20 17:48:11.953269 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.953258 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 17:48:11.953299 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.953275 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 17:48:11.955607 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.955583 2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 20 17:48:11.957388 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.957374 2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 20 17:48:11.957447 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.957404 2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 20 17:48:11.958375 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.958364 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 20 17:48:11.958428 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.958379 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 20 17:48:11.958428 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.958386 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 20 17:48:11.958428 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.958391 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 20 17:48:11.958428 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.958400 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 20 17:48:11.958428 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.958405 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 20 17:48:11.958428 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.958413 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 20 17:48:11.958428 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.958419 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 20 17:48:11.958428 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.958426 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 20 17:48:11.958428 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.958431 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 20 17:48:11.958650 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.958440 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 20 17:48:11.958650 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.958445 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 20 17:48:11.959800 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.959787 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 20 17:48:11.959843 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.959804 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 20 17:48:11.962913 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.962901 2583 server.go:1268] "Started kubelet" Jul 20 17:48:11.963003 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.962973 2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 20 17:48:11.963064 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.963010 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 20 17:48:11.963110 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.963090 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 20 17:48:11.963670 ip-10-0-141-129 systemd[1]: Started Kubernetes Kubelet. Jul 20 17:48:11.965224 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.965212 2583 server.go:310] "Adding debug handlers to kubelet server" Jul 20 17:48:11.966801 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.966784 2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 20 17:48:11.967417 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:11.967395 2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 20 17:48:11.971415 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.971400 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 17:48:11.971951 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.971938 2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 20 17:48:11.972056 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.971943 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 20 17:48:11.972185 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:11.972144 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-129.ec2.internal\" not found" Jul 20 17:48:11.972540 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.972523 2583 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 20 17:48:11.972540 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.972543 2583 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 20 17:48:11.972660 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.972561 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 20 17:48:11.972660 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.972627 2583 reconstruct.go:97] "Volume reconstruction finished" Jul 20 17:48:11.972660 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.972636 2583 reconciler.go:29] "Reconciler: start to sync state" Jul 20 17:48:11.972808 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.972688 2583 factory.go:55] Registering systemd factory Jul 20 17:48:11.972808 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.972699 2583 factory.go:223] Registration of the systemd container factory successfully Jul 20 17:48:11.972944 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.972929 2583 factory.go:153] Registering CRI-O factory Jul 20 17:48:11.972990 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.972948 2583 factory.go:223] Registration of the crio container factory successfully Jul 20 17:48:11.973033 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.973004 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 20 17:48:11.973033 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.973028 2583 factory.go:103] Registering Raw factory Jul 20 17:48:11.973123 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.973043 2583 manager.go:1196] Started watching for new ooms in manager Jul 20 17:48:11.973484 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.973470 2583 manager.go:349] Starting recovery of all containers Jul 20 17:48:11.983950 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:11.982600 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-129.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 20 17:48:11.983950 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:11.982660 2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-129.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 20 17:48:11.983950 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:11.982947 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 20 17:48:11.983950 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:11.982997 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 20 17:48:11.983950 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.983039 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-129.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 20 17:48:11.983950 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:11.982875 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-129.ec2.internal.18c410ae066024e3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-129.ec2.internal,UID:ip-10-0-141-129.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-129.ec2.internal,},FirstTimestamp:2026-07-20 17:48:11.962877155 +0000 UTC m=+0.334665720,LastTimestamp:2026-07-20 17:48:11.962877155 +0000 UTC m=+0.334665720,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-129.ec2.internal,}" Jul 20 17:48:11.985208 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.985190 2583 manager.go:354] Recovery completed Jul 20 17:48:11.989970 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.989954 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 17:48:11.991679 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.991655 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasSufficientMemory" Jul 20 17:48:11.991800 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.991701 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 17:48:11.991800 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.991719 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasSufficientPID" Jul 20 17:48:11.992320 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.992302 2583 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 20 17:48:11.992320 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.992316 2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 20 17:48:11.992402 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.992331 2583 state_mem.go:36] "Initialized new in-memory state store" Jul 20 17:48:11.992447 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.992307 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 17:48:11.993087 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.993074 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasSufficientMemory" Jul 20 17:48:11.993167 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.993102 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 17:48:11.993167 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.993115 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasSufficientPID" Jul 20 17:48:11.993470 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.993457 2583 policy_none.go:49] "None policy: Start" Jul 20 17:48:11.993520 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.993473 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 20 17:48:11.993520 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.993483 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 20 17:48:11.994558 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:11.994547 2583 policy_none.go:47] "Start" Jul 20 17:48:12.003445 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.003269 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-129.ec2.internal.18c410ae0817a1a8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-129.ec2.internal,UID:ip-10-0-141-129.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-129.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-129.ec2.internal,},FirstTimestamp:2026-07-20 17:48:11.9916794 +0000 UTC m=+0.363467969,LastTimestamp:2026-07-20 17:48:11.9916794 +0000 UTC m=+0.363467969,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-129.ec2.internal,}" Jul 20 17:48:12.015995 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.015930 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-129.ec2.internal.18c410ae08181135 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-129.ec2.internal,UID:ip-10-0-141-129.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-129.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-129.ec2.internal,},FirstTimestamp:2026-07-20 17:48:11.991707957 +0000 UTC m=+0.363496527,LastTimestamp:2026-07-20 17:48:11.991707957 +0000 UTC m=+0.363496527,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-129.ec2.internal,}" Jul 20 17:48:12.029344 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.029324 2583 manager.go:337] "Starting Device Plugin manager" Jul 20 17:48:12.029423 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.029365 2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 20 17:48:12.029423 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.029375 2583 server.go:87] "Starting device plugin registration server" Jul 20 17:48:12.029644 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.029632 2583 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 20 17:48:12.029699 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.029644 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 20 17:48:12.029775 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.029708 2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 20 17:48:12.029830 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.029806 2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 20 17:48:12.029830 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.029815 2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 20 17:48:12.030413 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.030386 2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 20 17:48:12.030487 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.030435 2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-129.ec2.internal\" not found" Jul 20 17:48:12.033695 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.033607 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-129.ec2.internal.18c410ae08185096 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-129.ec2.internal,UID:ip-10-0-141-129.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-129.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-129.ec2.internal,},FirstTimestamp:2026-07-20 17:48:11.991724182 +0000 UTC m=+0.363512749,LastTimestamp:2026-07-20 17:48:11.991724182 +0000 UTC m=+0.363512749,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-129.ec2.internal,}" Jul 20 17:48:12.044890 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.044805 2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-129.ec2.internal.18c410ae0817a1a8\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-129.ec2.internal.18c410ae0817a1a8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-129.ec2.internal,UID:ip-10-0-141-129.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-129.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-129.ec2.internal,},FirstTimestamp:2026-07-20 17:48:11.9916794 +0000 UTC m=+0.363467969,LastTimestamp:2026-07-20 17:48:11.993087734 +0000 UTC m=+0.364876302,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-129.ec2.internal,}" Jul 20 17:48:12.058585 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.058522 2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-129.ec2.internal.18c410ae08181135\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-129.ec2.internal.18c410ae08181135 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-129.ec2.internal,UID:ip-10-0-141-129.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-129.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-129.ec2.internal,},FirstTimestamp:2026-07-20 17:48:11.991707957 +0000 UTC m=+0.363496527,LastTimestamp:2026-07-20 17:48:11.993107798 +0000 UTC m=+0.364896364,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-129.ec2.internal,}" Jul 20 17:48:12.082029 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.081959 2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-129.ec2.internal.18c410ae08185096\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-129.ec2.internal.18c410ae08185096 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-129.ec2.internal,UID:ip-10-0-141-129.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-129.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-129.ec2.internal,},FirstTimestamp:2026-07-20 17:48:11.991724182 +0000 UTC m=+0.363512749,LastTimestamp:2026-07-20 17:48:11.993120998 +0000 UTC m=+0.364909565,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-129.ec2.internal,}" Jul 20 17:48:12.098113 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.098043 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-129.ec2.internal.18c410ae0a7c3a58 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-129.ec2.internal,UID:ip-10-0-141-129.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-141-129.ec2.internal,},FirstTimestamp:2026-07-20 17:48:12.03182652 +0000 UTC m=+0.403615072,LastTimestamp:2026-07-20 17:48:12.03182652 +0000 UTC m=+0.403615072,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-129.ec2.internal,}" Jul 20 17:48:12.121256 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.121232 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bnpgx" Jul 20 17:48:12.130333 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.130309 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 17:48:12.131254 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.131232 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasSufficientMemory" Jul 20 17:48:12.131365 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.131264 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 17:48:12.131365 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.131277 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasSufficientPID" Jul 20 17:48:12.131365 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.131308 2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-129.ec2.internal" Jul 20 17:48:12.137064 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.137044 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bnpgx" Jul 20 17:48:12.153816 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.153755 2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-129.ec2.internal" Jul 20 17:48:12.153816 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.153783 2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-129.ec2.internal\": node \"ip-10-0-141-129.ec2.internal\" not found" Jul 20 17:48:12.162664 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.162630 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 20 17:48:12.163900 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.163880 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 20 17:48:12.163992 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.163929 2583 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 20 17:48:12.163992 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.163954 2583 kubelet.go:2448] "Starting kubelet main sync loop" Jul 20 17:48:12.164087 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.163994 2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 20 17:48:12.167234 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.167218 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 17:48:12.188406 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.188389 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-129.ec2.internal\" not found" Jul 20 17:48:12.265102 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.264428 2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-129.ec2.internal"] Jul 20 17:48:12.265269 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.265142 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 17:48:12.266194 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.266172 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasSufficientMemory" Jul 20 17:48:12.266298 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.266207 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 17:48:12.266298 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.266223 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasSufficientPID" Jul 20 17:48:12.267350 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.267337 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 17:48:12.267500 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.267480 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal" Jul 20 17:48:12.267538 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.267518 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 17:48:12.268100 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.268079 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasSufficientMemory" Jul 20 17:48:12.268100 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.268085 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasSufficientMemory" Jul 20 17:48:12.268237 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.268106 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 17:48:12.268237 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.268121 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasSufficientPID" Jul 20 17:48:12.268237 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.268122 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 17:48:12.268237 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.268137 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasSufficientPID" Jul 20 17:48:12.269136 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.269122 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-129.ec2.internal" Jul 20 17:48:12.269184 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.269147 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 17:48:12.269814 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.269798 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasSufficientMemory" Jul 20 17:48:12.269903 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.269830 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 17:48:12.269903 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.269843 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeHasSufficientPID" Jul 20 17:48:12.274167 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.274153 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1cb3c65fdce6a2649d8a37538aaf404d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal\" (UID: \"1cb3c65fdce6a2649d8a37538aaf404d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal" Jul 20 17:48:12.274228 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.274176 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cb3c65fdce6a2649d8a37538aaf404d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal\" (UID: \"1cb3c65fdce6a2649d8a37538aaf404d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal" Jul 20 17:48:12.274228 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.274194 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7240dbae0531c45a7cd9f8a23f8cd206-config\") pod \"kube-apiserver-proxy-ip-10-0-141-129.ec2.internal\" (UID: \"7240dbae0531c45a7cd9f8a23f8cd206\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-129.ec2.internal" Jul 20 17:48:12.289288 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.289271 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-129.ec2.internal\" not found" Jul 20 17:48:12.298311 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.298293 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-129.ec2.internal\" not found" node="ip-10-0-141-129.ec2.internal" Jul 20 17:48:12.302857 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.302843 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-129.ec2.internal\" not found" node="ip-10-0-141-129.ec2.internal" Jul 20 17:48:12.374325 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.374298 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1cb3c65fdce6a2649d8a37538aaf404d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal\" (UID: \"1cb3c65fdce6a2649d8a37538aaf404d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal" Jul 20 17:48:12.374325 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.374327 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cb3c65fdce6a2649d8a37538aaf404d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal\" (UID: \"1cb3c65fdce6a2649d8a37538aaf404d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal" Jul 20 17:48:12.374471 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.374345 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7240dbae0531c45a7cd9f8a23f8cd206-config\") pod \"kube-apiserver-proxy-ip-10-0-141-129.ec2.internal\" (UID: \"7240dbae0531c45a7cd9f8a23f8cd206\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-129.ec2.internal" Jul 20 17:48:12.374471 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.374396 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7240dbae0531c45a7cd9f8a23f8cd206-config\") pod \"kube-apiserver-proxy-ip-10-0-141-129.ec2.internal\" (UID: \"7240dbae0531c45a7cd9f8a23f8cd206\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-129.ec2.internal" Jul 20 17:48:12.374471 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.374424 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cb3c65fdce6a2649d8a37538aaf404d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal\" (UID: \"1cb3c65fdce6a2649d8a37538aaf404d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal" Jul 20 17:48:12.374471 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.374396 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1cb3c65fdce6a2649d8a37538aaf404d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal\" (UID: \"1cb3c65fdce6a2649d8a37538aaf404d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal" Jul 20 17:48:12.389781 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.389759 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-129.ec2.internal\" not found" Jul 20 17:48:12.489933 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.489870 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-129.ec2.internal\" not found" Jul 20 17:48:12.590693 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.590669 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-129.ec2.internal\" not found" Jul 20 17:48:12.600877 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.600854 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal" Jul 20 17:48:12.605464 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.605444 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-129.ec2.internal" Jul 20 17:48:12.690970 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.690939 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-129.ec2.internal\" not found" Jul 20 17:48:12.791548 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.791485 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-129.ec2.internal\" not found" Jul 20 17:48:12.892015 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.891980 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-129.ec2.internal\" not found" Jul 20 17:48:12.916146 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.916126 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 20 17:48:12.916539 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:12.916264 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 20 17:48:12.992422 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:12.992398 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-129.ec2.internal\" not found" Jul 20 17:48:13.009271 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.009253 2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 17:48:13.072213 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.072154 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal" Jul 20 17:48:13.099384 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:13.099354 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb3c65fdce6a2649d8a37538aaf404d.slice/crio-6d722bdd93a4f4108f3ab78ce5bb7462474a9d87f483a8728a84ca56d8c55ffd WatchSource:0}: Error finding container 6d722bdd93a4f4108f3ab78ce5bb7462474a9d87f483a8728a84ca56d8c55ffd: Status 404 returned error can't find the container with id 6d722bdd93a4f4108f3ab78ce5bb7462474a9d87f483a8728a84ca56d8c55ffd Jul 20 17:48:13.099835 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:13.099808 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7240dbae0531c45a7cd9f8a23f8cd206.slice/crio-3425d7268791ddce612ee6a3df0b079ccb95b4b34298ef1e42a4fa96f2173a33 WatchSource:0}: Error finding container 3425d7268791ddce612ee6a3df0b079ccb95b4b34298ef1e42a4fa96f2173a33: Status 404 returned error can't find the container with id 3425d7268791ddce612ee6a3df0b079ccb95b4b34298ef1e42a4fa96f2173a33 Jul 20 17:48:13.105054 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.105041 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 17:48:13.122203 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.122182 2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 17:48:13.131223 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.131200 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 17:48:13.131968 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.131956 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-129.ec2.internal" Jul 20 17:48:13.140305 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.140279 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 17:43:12 +0000 UTC" deadline="2028-04-07 11:07:15.583593128 +0000 UTC" Jul 20 17:48:13.140305 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.140303 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15041h19m2.443292568s" Jul 20 17:48:13.168169 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.168121 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal" event={"ID":"1cb3c65fdce6a2649d8a37538aaf404d","Type":"ContainerStarted","Data":"6d722bdd93a4f4108f3ab78ce5bb7462474a9d87f483a8728a84ca56d8c55ffd"} Jul 20 17:48:13.169165 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.169138 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-129.ec2.internal" event={"ID":"7240dbae0531c45a7cd9f8a23f8cd206","Type":"ContainerStarted","Data":"3425d7268791ddce612ee6a3df0b079ccb95b4b34298ef1e42a4fa96f2173a33"} Jul 20 17:48:13.173254 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.173242 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 17:48:13.453521 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.453447 2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 17:48:13.930634 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.930553 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 17:48:13.955376 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.955351 2583 apiserver.go:52] "Watching apiserver" Jul 20 17:48:13.964352 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.964325 2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 20 17:48:13.965593 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.965567 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-v9j9l","kube-system/kube-apiserver-proxy-ip-10-0-141-129.ec2.internal","openshift-cluster-node-tuning-operator/tuned-55xlw","openshift-image-registry/node-ca-7jrr6","openshift-multus/multus-q9g6b","openshift-multus/network-metrics-daemon-r24gs","openshift-network-diagnostics/network-check-target-8rmfb","openshift-ovn-kubernetes/ovnkube-node-m5tpx","kube-system/global-pull-secret-syncer-rshx6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk","openshift-dns/node-resolver-qgk6s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal","openshift-multus/multus-additional-cni-plugins-2xsj4","openshift-network-operator/iptables-alerter-75t9c"] Jul 20 17:48:13.967014 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.966996 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:13.967103 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:13.967075 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rmfb" podUID="e8d261a8-44f9-4a08-b932-236b2306b4f7" Jul 20 17:48:13.968722 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.968706 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:13.972703 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.970165 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:13.972703 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.971669 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7jrr6" Jul 20 17:48:13.972703 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.972620 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q9g6b" Jul 20 17:48:13.973037 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.973000 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 20 17:48:13.974039 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.974012 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:13.974417 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:13.974393 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r24gs" podUID="76af6cb3-7b9d-4993-a5c8-24afeb42b60b" Jul 20 17:48:13.975600 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.975584 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:13.975687 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:13.975664 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rshx6" podUID="21af06f0-d59b-4e61-84fb-57425520c8c3" Jul 20 17:48:13.976784 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.976765 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v9j9l" Jul 20 17:48:13.978079 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.978057 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-75t9c" Jul 20 17:48:13.979445 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.979428 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:13.980661 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.980646 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qgk6s" Jul 20 17:48:13.981549 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.981531 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-host\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:13.981647 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.981555 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h2j2\" (UniqueName: \"kubernetes.io/projected/0ff18797-615b-4cc4-82f2-3621080f562e-kube-api-access-5h2j2\") pod \"node-ca-7jrr6\" (UID: \"0ff18797-615b-4cc4-82f2-3621080f562e\") " pod="openshift-image-registry/node-ca-7jrr6" Jul 20 17:48:13.981647 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.981571 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a265dbdd-eaa3-465c-a32b-022b3b94f594-multus-daemon-config\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:13.981647 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.981587 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-host-run-multus-certs\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:13.981647 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.981603 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-cni-binary-copy\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:13.981836 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.981647 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-etc-sysctl-d\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:13.981836 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.981669 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-run\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:13.981836 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.981694 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-lib-modules\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:13.981836 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.981716 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a819ab4d-1473-490e-831c-733e4d0a729b-tmp\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:13.981836 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.981769 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a265dbdd-eaa3-465c-a32b-022b3b94f594-cni-binary-copy\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:13.981836 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.981808 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-host-var-lib-cni-multus\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:13.981836 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.981835 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-multus-conf-dir\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:13.982160 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.981859 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:13.982160 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.981882 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-system-cni-dir\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:13.982160 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.981936 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-multus-cni-dir\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:13.982160 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.981950 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-etc-kubernetes\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:13.982160 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.981988 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-var-lib-kubelet\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:13.982160 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982008 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-host-var-lib-cni-bin\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:13.982160 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982023 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpqbk\" (UniqueName: \"kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk\") pod \"network-check-target-8rmfb\" (UID: \"e8d261a8-44f9-4a08-b932-236b2306b4f7\") " pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:13.982160 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982042 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-etc-modprobe-d\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:13.982160 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982064 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-etc-systemd\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:13.982160 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982085 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ff18797-615b-4cc4-82f2-3621080f562e-serviceca\") pod \"node-ca-7jrr6\" (UID: \"0ff18797-615b-4cc4-82f2-3621080f562e\") " pod="openshift-image-registry/node-ca-7jrr6" Jul 20 17:48:13.982160 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982126 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-multus-socket-dir-parent\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:13.982671 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982189 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-host-run-netns\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:13.982671 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982215 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-os-release\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:13.982671 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982240 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgmht\" (UniqueName: \"kubernetes.io/projected/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-kube-api-access-rgmht\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:13.982671 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982264 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-etc-sysconfig\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:13.982671 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982284 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ff18797-615b-4cc4-82f2-3621080f562e-host\") pod \"node-ca-7jrr6\" (UID: \"0ff18797-615b-4cc4-82f2-3621080f562e\") " pod="openshift-image-registry/node-ca-7jrr6" Jul 20 17:48:13.982671 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982299 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:13.982671 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982306 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-cnibin\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:13.982671 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982327 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-host-var-lib-kubelet\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:13.982671 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982351 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4htd\" (UniqueName: \"kubernetes.io/projected/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-kube-api-access-p4htd\") pod \"network-metrics-daemon-r24gs\" (UID: \"76af6cb3-7b9d-4993-a5c8-24afeb42b60b\") " pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:13.982671 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982374 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-system-cni-dir\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:13.982671 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982396 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-etc-sysctl-conf\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:13.982671 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982416 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-sys\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:13.982671 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982438 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljmmf\" (UniqueName: \"kubernetes.io/projected/a819ab4d-1473-490e-831c-733e4d0a729b-kube-api-access-ljmmf\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:13.982671 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982459 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-os-release\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:13.982671 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982481 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-host-run-k8s-cni-cncf-io\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:13.982671 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982503 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-etc-kubernetes\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:13.982671 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982528 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tr6c\" (UniqueName: \"kubernetes.io/projected/a265dbdd-eaa3-465c-a32b-022b3b94f594-kube-api-access-8tr6c\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:13.983238 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982551 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-cnibin\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:13.983238 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982577 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:13.983238 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982601 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:13.983238 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982626 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a819ab4d-1473-490e-831c-733e4d0a729b-etc-tuned\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:13.983238 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982649 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-hostroot\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:13.983238 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.982672 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs\") pod \"network-metrics-daemon-r24gs\" (UID: \"76af6cb3-7b9d-4993-a5c8-24afeb42b60b\") " pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:13.989393 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.989360 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 20 17:48:13.989546 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.989496 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 20 17:48:13.993662 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.993631 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 20 17:48:13.993793 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.993714 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 20 17:48:13.993950 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.993907 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5qspx\"" Jul 20 17:48:13.994130 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.994074 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vfmwj\"" Jul 20 17:48:13.994473 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.994457 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 20 17:48:13.994989 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.994973 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 20 17:48:13.996811 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.996791 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 20 17:48:13.996903 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:13.996802 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7vvt7\"" Jul 20 17:48:14.000851 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.000821 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 20 17:48:14.000945 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.000861 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n82g4\"" Jul 20 17:48:14.000999 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.000963 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 20 17:48:14.001289 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.001269 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 20 17:48:14.001496 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.001482 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 20 17:48:14.002084 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.002061 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 20 17:48:14.002522 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.002359 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 20 17:48:14.002522 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.002369 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 20 17:48:14.002522 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.002411 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 20 17:48:14.002689 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.002579 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f8trf\"" Jul 20 17:48:14.003145 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.003129 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 20 17:48:14.003388 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.003345 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 20 17:48:14.003388 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.003362 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 20 17:48:14.003577 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.003444 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 20 17:48:14.003577 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.003548 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 20 17:48:14.004393 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.004375 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gxl2d\"" Jul 20 17:48:14.004492 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.004407 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 20 17:48:14.004569 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.004547 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wxztv\"" Jul 20 17:48:14.004718 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.004596 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 20 17:48:14.004718 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.004682 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-52jkk\"" Jul 20 17:48:14.005261 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.005244 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 20 17:48:14.013050 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.013032 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 20 17:48:14.021581 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.021561 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8zzvs\"" Jul 20 17:48:14.022150 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.022129 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 20 17:48:14.023943 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.023922 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 20 17:48:14.024017 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.023958 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 20 17:48:14.059876 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.059854 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 17:48:14.073497 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.073477 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 20 17:48:14.083863 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.083834 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-etc-kubernetes\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.083968 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.083891 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tr6c\" (UniqueName: \"kubernetes.io/projected/a265dbdd-eaa3-465c-a32b-022b3b94f594-kube-api-access-8tr6c\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.083968 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.083925 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-etc-kubernetes\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.084447 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.084426 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:14.086199 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.083936 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:14.086199 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.084644 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a819ab4d-1473-490e-831c-733e4d0a729b-etc-tuned\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.086199 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.084673 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-hostroot\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.086199 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.084703 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs\") pod \"network-metrics-daemon-r24gs\" (UID: \"76af6cb3-7b9d-4993-a5c8-24afeb42b60b\") " pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:14.086199 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.084755 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-host\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.086199 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.084788 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-run-netns\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.086199 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.084812 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-run-openvswitch\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.086199 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.084845 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a265dbdd-eaa3-465c-a32b-022b3b94f594-multus-daemon-config\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.086199 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.084874 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-host-run-multus-certs\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.086199 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.084906 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-cni-binary-copy\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:14.086199 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.084934 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-etc-sysctl-d\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.086199 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.084956 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-run\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.086199 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.084982 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a819ab4d-1473-490e-831c-733e4d0a729b-tmp\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.086199 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085008 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-cni-bin\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.086199 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085037 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:14.086199 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085073 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knrg4\" (UniqueName: \"kubernetes.io/projected/f5bf65ad-15fc-46c4-a109-34b8132e2431-kube-api-access-knrg4\") pod \"iptables-alerter-75t9c\" (UID: \"f5bf65ad-15fc-46c4-a109-34b8132e2431\") " pod="openshift-network-operator/iptables-alerter-75t9c" Jul 20 17:48:14.086199 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085098 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/167b1498-09ad-4ab3-84e9-8787235afb26-tmp-dir\") pod \"node-resolver-qgk6s\" (UID: \"167b1498-09ad-4ab3-84e9-8787235afb26\") " pod="openshift-dns/node-resolver-qgk6s" Jul 20 17:48:14.087054 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085129 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mskpj\" (UniqueName: \"kubernetes.io/projected/167b1498-09ad-4ab3-84e9-8787235afb26-kube-api-access-mskpj\") pod \"node-resolver-qgk6s\" (UID: \"167b1498-09ad-4ab3-84e9-8787235afb26\") " pod="openshift-dns/node-resolver-qgk6s" Jul 20 17:48:14.087054 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085159 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-system-cni-dir\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.087054 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085187 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-multus-cni-dir\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.087054 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085215 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-etc-kubernetes\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.087054 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085238 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-var-lib-kubelet\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.087054 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085266 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7194a6f0-d226-438f-8445-2db548350c5b-konnectivity-ca\") pod \"konnectivity-agent-v9j9l\" (UID: \"7194a6f0-d226-438f-8445-2db548350c5b\") " pod="kube-system/konnectivity-agent-v9j9l" Jul 20 17:48:14.087054 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085293 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/167b1498-09ad-4ab3-84e9-8787235afb26-hosts-file\") pod \"node-resolver-qgk6s\" (UID: \"167b1498-09ad-4ab3-84e9-8787235afb26\") " pod="openshift-dns/node-resolver-qgk6s" Jul 20 17:48:14.087054 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085322 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-log-socket\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.087054 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085351 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-host-var-lib-cni-bin\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.087054 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085377 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqbk\" (UniqueName: \"kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk\") pod \"network-check-target-8rmfb\" (UID: \"e8d261a8-44f9-4a08-b932-236b2306b4f7\") " pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:14.087054 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085407 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-etc-modprobe-d\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.087054 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085437 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-etc-systemd\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.087054 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085465 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-etc-openvswitch\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.087054 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085497 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.087054 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085521 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/83a7f11b-2c9a-45ed-923c-4757f07212c9-env-overrides\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.087054 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085556 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-os-release\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:14.087054 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085587 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgmht\" (UniqueName: \"kubernetes.io/projected/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-kube-api-access-rgmht\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:14.087821 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085620 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-etc-sysconfig\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.087821 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085649 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ff18797-615b-4cc4-82f2-3621080f562e-host\") pod \"node-ca-7jrr6\" (UID: \"0ff18797-615b-4cc4-82f2-3621080f562e\") " pod="openshift-image-registry/node-ca-7jrr6" Jul 20 17:48:14.087821 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085679 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.087821 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085704 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/83a7f11b-2c9a-45ed-923c-4757f07212c9-ovnkube-script-lib\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.087821 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085753 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-system-cni-dir\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:14.087821 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085784 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-etc-sysctl-conf\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.087821 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085814 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljmmf\" (UniqueName: \"kubernetes.io/projected/a819ab4d-1473-490e-831c-733e4d0a729b-kube-api-access-ljmmf\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.087821 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085844 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5bf65ad-15fc-46c4-a109-34b8132e2431-host-slash\") pod \"iptables-alerter-75t9c\" (UID: \"f5bf65ad-15fc-46c4-a109-34b8132e2431\") " pod="openshift-network-operator/iptables-alerter-75t9c" Jul 20 17:48:14.087821 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085869 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-run-ovn\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.087821 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085897 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-os-release\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.087821 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085927 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-host-run-k8s-cni-cncf-io\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.087821 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085956 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-cnibin\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:14.087821 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.085986 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:14.087821 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086013 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2j2\" (UniqueName: \"kubernetes.io/projected/0ff18797-615b-4cc4-82f2-3621080f562e-kube-api-access-5h2j2\") pod \"node-ca-7jrr6\" (UID: \"0ff18797-615b-4cc4-82f2-3621080f562e\") " pod="openshift-image-registry/node-ca-7jrr6" Jul 20 17:48:14.087821 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086043 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-run-systemd\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.087821 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086126 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret\") pod \"global-pull-secret-syncer-rshx6\" (UID: \"21af06f0-d59b-4e61-84fb-57425520c8c3\") " pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:14.087821 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086183 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-run\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086223 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-host\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086246 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-system-cni-dir\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086417 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-os-release\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086473 2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086484 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-hostroot\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086476 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-system-cni-dir\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086528 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-etc-kubernetes\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086565 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-etc-systemd\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086577 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-host-run-k8s-cni-cncf-io\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086589 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-etc-sysconfig\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086620 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-var-lib-kubelet\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086745 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-etc-sysctl-conf\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086766 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086880 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-host-run-multus-certs\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.086984 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-cni-binary-copy\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087085 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-host-var-lib-cni-bin\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087063 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087140 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ff18797-615b-4cc4-82f2-3621080f562e-host\") pod \"node-ca-7jrr6\" (UID: \"0ff18797-615b-4cc4-82f2-3621080f562e\") " pod="openshift-image-registry/node-ca-7jrr6" Jul 20 17:48:14.088504 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:14.087162 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:14.089346 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087195 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a0df65aa-855d-41d5-9a22-3368606a7700-registration-dir\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.089346 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087274 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a265dbdd-eaa3-465c-a32b-022b3b94f594-multus-daemon-config\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.089346 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087291 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-etc-sysctl-d\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.089346 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087283 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-os-release\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:14.089346 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087308 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-etc-modprobe-d\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.089346 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087451 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-cnibin\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:14.089346 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087514 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a0df65aa-855d-41d5-9a22-3368606a7700-device-dir\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.089346 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087544 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-multus-cni-dir\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.089346 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:14.087593 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs podName:76af6cb3-7b9d-4993-a5c8-24afeb42b60b nodeName:}" failed. No retries permitted until 2026-07-20 17:48:14.587539076 +0000 UTC m=+2.959327645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs") pod "network-metrics-daemon-r24gs" (UID: "76af6cb3-7b9d-4993-a5c8-24afeb42b60b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:14.089346 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087627 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/83a7f11b-2c9a-45ed-923c-4757f07212c9-ovn-node-metrics-cert\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.089346 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087677 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-lib-modules\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.089346 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087764 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-systemd-units\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.089346 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087817 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a265dbdd-eaa3-465c-a32b-022b3b94f594-cni-binary-copy\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.089346 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087820 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-lib-modules\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.089346 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087866 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-host-var-lib-cni-multus\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.089346 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087917 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-multus-conf-dir\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.090058 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.087957 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21af06f0-d59b-4e61-84fb-57425520c8c3-kubelet-config\") pod \"global-pull-secret-syncer-rshx6\" (UID: \"21af06f0-d59b-4e61-84fb-57425520c8c3\") " pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:14.090058 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.088018 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-host-var-lib-cni-multus\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.090058 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.088053 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-kubelet\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.090058 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.088106 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-var-lib-openvswitch\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.090058 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.088587 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-multus-conf-dir\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.090058 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.088618 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-node-log\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.090058 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.088675 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a0df65aa-855d-41d5-9a22-3368606a7700-socket-dir\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.090058 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.088711 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbtdr\" (UniqueName: \"kubernetes.io/projected/a0df65aa-855d-41d5-9a22-3368606a7700-kube-api-access-dbtdr\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.090058 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.088773 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/83a7f11b-2c9a-45ed-923c-4757f07212c9-ovnkube-config\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.090058 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.088807 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ff18797-615b-4cc4-82f2-3621080f562e-serviceca\") pod \"node-ca-7jrr6\" (UID: \"0ff18797-615b-4cc4-82f2-3621080f562e\") " pod="openshift-image-registry/node-ca-7jrr6" Jul 20 17:48:14.090058 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.088809 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a265dbdd-eaa3-465c-a32b-022b3b94f594-cni-binary-copy\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.090058 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.088833 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a0df65aa-855d-41d5-9a22-3368606a7700-kubelet-dir\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.090058 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.088894 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-multus-socket-dir-parent\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.090058 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.088925 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-host-run-netns\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.090058 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.088952 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7194a6f0-d226-438f-8445-2db548350c5b-agent-certs\") pod \"konnectivity-agent-v9j9l\" (UID: \"7194a6f0-d226-438f-8445-2db548350c5b\") " pod="kube-system/konnectivity-agent-v9j9l" Jul 20 17:48:14.090058 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.088989 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a0df65aa-855d-41d5-9a22-3368606a7700-etc-selinux\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.090058 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.089015 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a0df65aa-855d-41d5-9a22-3368606a7700-sys-fs\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.090951 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.089041 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-slash\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.090951 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.089064 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-cni-netd\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.090951 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.089098 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87k9h\" (UniqueName: \"kubernetes.io/projected/83a7f11b-2c9a-45ed-923c-4757f07212c9-kube-api-access-87k9h\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.090951 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.089126 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-cnibin\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.090951 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.089148 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-host-run-netns\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.090951 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.089153 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-host-var-lib-kubelet\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.090951 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.089176 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-cnibin\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.090951 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.089178 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-host-var-lib-kubelet\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.090951 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.089202 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4htd\" (UniqueName: \"kubernetes.io/projected/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-kube-api-access-p4htd\") pod \"network-metrics-daemon-r24gs\" (UID: \"76af6cb3-7b9d-4993-a5c8-24afeb42b60b\") " pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:14.090951 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.089231 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-sys\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.090951 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.089232 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a265dbdd-eaa3-465c-a32b-022b3b94f594-multus-socket-dir-parent\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.090951 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.089266 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21af06f0-d59b-4e61-84fb-57425520c8c3-dbus\") pod \"global-pull-secret-syncer-rshx6\" (UID: \"21af06f0-d59b-4e61-84fb-57425520c8c3\") " pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:14.090951 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.089268 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ff18797-615b-4cc4-82f2-3621080f562e-serviceca\") pod \"node-ca-7jrr6\" (UID: \"0ff18797-615b-4cc4-82f2-3621080f562e\") " pod="openshift-image-registry/node-ca-7jrr6" Jul 20 17:48:14.090951 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.089295 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5bf65ad-15fc-46c4-a109-34b8132e2431-iptables-alerter-script\") pod \"iptables-alerter-75t9c\" (UID: \"f5bf65ad-15fc-46c4-a109-34b8132e2431\") " pod="openshift-network-operator/iptables-alerter-75t9c" Jul 20 17:48:14.090951 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.089301 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a819ab4d-1473-490e-831c-733e4d0a729b-sys\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.090951 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.090026 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a819ab4d-1473-490e-831c-733e4d0a729b-etc-tuned\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.090951 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.090099 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a819ab4d-1473-490e-831c-733e4d0a729b-tmp\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.097820 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.097791 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgmht\" (UniqueName: \"kubernetes.io/projected/e8583b5a-bbcd-44ba-9c03-2a14cfa40f19-kube-api-access-rgmht\") pod \"multus-additional-cni-plugins-2xsj4\" (UID: \"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19\") " pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:14.101879 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.101862 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tr6c\" (UniqueName: \"kubernetes.io/projected/a265dbdd-eaa3-465c-a32b-022b3b94f594-kube-api-access-8tr6c\") pod \"multus-q9g6b\" (UID: \"a265dbdd-eaa3-465c-a32b-022b3b94f594\") " pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.104682 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:14.104662 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 17:48:14.104781 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:14.104687 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 17:48:14.104781 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:14.104704 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gpqbk for pod openshift-network-diagnostics/network-check-target-8rmfb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:14.104859 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:14.104793 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk podName:e8d261a8-44f9-4a08-b932-236b2306b4f7 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:14.604778857 +0000 UTC m=+2.976567430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gpqbk" (UniqueName: "kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk") pod "network-check-target-8rmfb" (UID: "e8d261a8-44f9-4a08-b932-236b2306b4f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:14.106716 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.106697 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h2j2\" (UniqueName: \"kubernetes.io/projected/0ff18797-615b-4cc4-82f2-3621080f562e-kube-api-access-5h2j2\") pod \"node-ca-7jrr6\" (UID: \"0ff18797-615b-4cc4-82f2-3621080f562e\") " pod="openshift-image-registry/node-ca-7jrr6" Jul 20 17:48:14.111141 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.111122 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4htd\" (UniqueName: \"kubernetes.io/projected/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-kube-api-access-p4htd\") pod \"network-metrics-daemon-r24gs\" (UID: \"76af6cb3-7b9d-4993-a5c8-24afeb42b60b\") " pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:14.117547 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.117531 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljmmf\" (UniqueName: \"kubernetes.io/projected/a819ab4d-1473-490e-831c-733e4d0a729b-kube-api-access-ljmmf\") pod \"tuned-55xlw\" (UID: \"a819ab4d-1473-490e-831c-733e4d0a729b\") " pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.190055 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.189968 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbtdr\" (UniqueName: \"kubernetes.io/projected/a0df65aa-855d-41d5-9a22-3368606a7700-kube-api-access-dbtdr\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.190055 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190004 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/83a7f11b-2c9a-45ed-923c-4757f07212c9-ovnkube-config\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.190055 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190028 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a0df65aa-855d-41d5-9a22-3368606a7700-kubelet-dir\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.190055 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190054 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7194a6f0-d226-438f-8445-2db548350c5b-agent-certs\") pod \"konnectivity-agent-v9j9l\" (UID: \"7194a6f0-d226-438f-8445-2db548350c5b\") " pod="kube-system/konnectivity-agent-v9j9l" Jul 20 17:48:14.190408 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190120 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a0df65aa-855d-41d5-9a22-3368606a7700-etc-selinux\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.190408 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190161 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a0df65aa-855d-41d5-9a22-3368606a7700-sys-fs\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.190408 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190185 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-slash\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.190408 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190191 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a0df65aa-855d-41d5-9a22-3368606a7700-kubelet-dir\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.190408 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190257 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a0df65aa-855d-41d5-9a22-3368606a7700-sys-fs\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.190408 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190319 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a0df65aa-855d-41d5-9a22-3368606a7700-etc-selinux\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.190408 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190343 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-slash\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.190408 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190368 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-cni-netd\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.190408 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190396 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87k9h\" (UniqueName: \"kubernetes.io/projected/83a7f11b-2c9a-45ed-923c-4757f07212c9-kube-api-access-87k9h\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.190867 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190422 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21af06f0-d59b-4e61-84fb-57425520c8c3-dbus\") pod \"global-pull-secret-syncer-rshx6\" (UID: \"21af06f0-d59b-4e61-84fb-57425520c8c3\") " pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:14.190867 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190455 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-cni-netd\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.190867 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190491 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21af06f0-d59b-4e61-84fb-57425520c8c3-dbus\") pod \"global-pull-secret-syncer-rshx6\" (UID: \"21af06f0-d59b-4e61-84fb-57425520c8c3\") " pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:14.190867 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190527 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5bf65ad-15fc-46c4-a109-34b8132e2431-iptables-alerter-script\") pod \"iptables-alerter-75t9c\" (UID: \"f5bf65ad-15fc-46c4-a109-34b8132e2431\") " pod="openshift-network-operator/iptables-alerter-75t9c" Jul 20 17:48:14.190867 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190590 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-run-netns\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.190867 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190642 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-run-openvswitch\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.190867 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190683 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-cni-bin\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.190867 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190689 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-run-netns\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.190867 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190753 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-run-openvswitch\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.190867 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190772 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-cni-bin\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.190867 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190780 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/83a7f11b-2c9a-45ed-923c-4757f07212c9-ovnkube-config\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.190867 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190795 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knrg4\" (UniqueName: \"kubernetes.io/projected/f5bf65ad-15fc-46c4-a109-34b8132e2431-kube-api-access-knrg4\") pod \"iptables-alerter-75t9c\" (UID: \"f5bf65ad-15fc-46c4-a109-34b8132e2431\") " pod="openshift-network-operator/iptables-alerter-75t9c" Jul 20 17:48:14.190867 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190837 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/167b1498-09ad-4ab3-84e9-8787235afb26-tmp-dir\") pod \"node-resolver-qgk6s\" (UID: \"167b1498-09ad-4ab3-84e9-8787235afb26\") " pod="openshift-dns/node-resolver-qgk6s" Jul 20 17:48:14.190867 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190864 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mskpj\" (UniqueName: \"kubernetes.io/projected/167b1498-09ad-4ab3-84e9-8787235afb26-kube-api-access-mskpj\") pod \"node-resolver-qgk6s\" (UID: \"167b1498-09ad-4ab3-84e9-8787235afb26\") " pod="openshift-dns/node-resolver-qgk6s" Jul 20 17:48:14.191457 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190895 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7194a6f0-d226-438f-8445-2db548350c5b-konnectivity-ca\") pod \"konnectivity-agent-v9j9l\" (UID: \"7194a6f0-d226-438f-8445-2db548350c5b\") " pod="kube-system/konnectivity-agent-v9j9l" Jul 20 17:48:14.191457 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190919 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/167b1498-09ad-4ab3-84e9-8787235afb26-hosts-file\") pod \"node-resolver-qgk6s\" (UID: \"167b1498-09ad-4ab3-84e9-8787235afb26\") " pod="openshift-dns/node-resolver-qgk6s" Jul 20 17:48:14.191457 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190940 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-log-socket\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.191457 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.190979 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-etc-openvswitch\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.191457 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191004 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.191457 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191014 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5bf65ad-15fc-46c4-a109-34b8132e2431-iptables-alerter-script\") pod \"iptables-alerter-75t9c\" (UID: \"f5bf65ad-15fc-46c4-a109-34b8132e2431\") " pod="openshift-network-operator/iptables-alerter-75t9c" Jul 20 17:48:14.191457 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191060 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/83a7f11b-2c9a-45ed-923c-4757f07212c9-env-overrides\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.191457 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191086 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/167b1498-09ad-4ab3-84e9-8787235afb26-hosts-file\") pod \"node-resolver-qgk6s\" (UID: \"167b1498-09ad-4ab3-84e9-8787235afb26\") " pod="openshift-dns/node-resolver-qgk6s" Jul 20 17:48:14.191457 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191103 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.191457 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191129 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-etc-openvswitch\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.191457 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191138 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-log-socket\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.191457 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191104 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/167b1498-09ad-4ab3-84e9-8787235afb26-tmp-dir\") pod \"node-resolver-qgk6s\" (UID: \"167b1498-09ad-4ab3-84e9-8787235afb26\") " pod="openshift-dns/node-resolver-qgk6s" Jul 20 17:48:14.191457 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191138 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/83a7f11b-2c9a-45ed-923c-4757f07212c9-ovnkube-script-lib\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.191457 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191195 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5bf65ad-15fc-46c4-a109-34b8132e2431-host-slash\") pod \"iptables-alerter-75t9c\" (UID: \"f5bf65ad-15fc-46c4-a109-34b8132e2431\") " pod="openshift-network-operator/iptables-alerter-75t9c" Jul 20 17:48:14.191457 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191220 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-run-ovn\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.191457 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191273 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-run-ovn\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.191457 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191278 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.192292 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191317 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5bf65ad-15fc-46c4-a109-34b8132e2431-host-slash\") pod \"iptables-alerter-75t9c\" (UID: \"f5bf65ad-15fc-46c4-a109-34b8132e2431\") " pod="openshift-network-operator/iptables-alerter-75t9c" Jul 20 17:48:14.192292 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191432 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.192292 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191459 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-run-systemd\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.192292 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191487 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret\") pod \"global-pull-secret-syncer-rshx6\" (UID: \"21af06f0-d59b-4e61-84fb-57425520c8c3\") " pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:14.192292 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191511 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a0df65aa-855d-41d5-9a22-3368606a7700-registration-dir\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.192292 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191528 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/83a7f11b-2c9a-45ed-923c-4757f07212c9-env-overrides\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.192292 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191533 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a0df65aa-855d-41d5-9a22-3368606a7700-device-dir\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.192292 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191558 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/83a7f11b-2c9a-45ed-923c-4757f07212c9-ovn-node-metrics-cert\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.192292 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191584 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-systemd-units\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.192292 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191586 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-run-systemd\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.192292 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191598 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a0df65aa-855d-41d5-9a22-3368606a7700-registration-dir\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.192292 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191608 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a0df65aa-855d-41d5-9a22-3368606a7700-device-dir\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.192292 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191611 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21af06f0-d59b-4e61-84fb-57425520c8c3-kubelet-config\") pod \"global-pull-secret-syncer-rshx6\" (UID: \"21af06f0-d59b-4e61-84fb-57425520c8c3\") " pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:14.192292 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:14.191625 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:14.192292 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191647 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-systemd-units\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.192292 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191655 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21af06f0-d59b-4e61-84fb-57425520c8c3-kubelet-config\") pod \"global-pull-secret-syncer-rshx6\" (UID: \"21af06f0-d59b-4e61-84fb-57425520c8c3\") " pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:14.192292 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191660 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-kubelet\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.193094 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:14.191687 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret podName:21af06f0-d59b-4e61-84fb-57425520c8c3 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:14.691666526 +0000 UTC m=+3.063455091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret") pod "global-pull-secret-syncer-rshx6" (UID: "21af06f0-d59b-4e61-84fb-57425520c8c3") : object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:14.193094 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191703 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-host-kubelet\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.193094 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191752 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-var-lib-openvswitch\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.193094 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191773 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-node-log\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.193094 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191778 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/83a7f11b-2c9a-45ed-923c-4757f07212c9-ovnkube-script-lib\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.193094 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191787 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a0df65aa-855d-41d5-9a22-3368606a7700-socket-dir\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.193094 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191890 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-node-log\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.193094 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191858 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83a7f11b-2c9a-45ed-923c-4757f07212c9-var-lib-openvswitch\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.193094 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.191929 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a0df65aa-855d-41d5-9a22-3368606a7700-socket-dir\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.193094 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.192053 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7194a6f0-d226-438f-8445-2db548350c5b-konnectivity-ca\") pod \"konnectivity-agent-v9j9l\" (UID: \"7194a6f0-d226-438f-8445-2db548350c5b\") " pod="kube-system/konnectivity-agent-v9j9l" Jul 20 17:48:14.193094 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.192955 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7194a6f0-d226-438f-8445-2db548350c5b-agent-certs\") pod \"konnectivity-agent-v9j9l\" (UID: \"7194a6f0-d226-438f-8445-2db548350c5b\") " pod="kube-system/konnectivity-agent-v9j9l" Jul 20 17:48:14.193895 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.193857 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/83a7f11b-2c9a-45ed-923c-4757f07212c9-ovn-node-metrics-cert\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.208632 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.208607 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9ppkw" Jul 20 17:48:14.227573 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.227548 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbtdr\" (UniqueName: \"kubernetes.io/projected/a0df65aa-855d-41d5-9a22-3368606a7700-kube-api-access-dbtdr\") pod \"aws-ebs-csi-driver-node-567wk\" (UID: \"a0df65aa-855d-41d5-9a22-3368606a7700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.228193 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.228170 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knrg4\" (UniqueName: \"kubernetes.io/projected/f5bf65ad-15fc-46c4-a109-34b8132e2431-kube-api-access-knrg4\") pod \"iptables-alerter-75t9c\" (UID: \"f5bf65ad-15fc-46c4-a109-34b8132e2431\") " pod="openshift-network-operator/iptables-alerter-75t9c" Jul 20 17:48:14.229384 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.229363 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mskpj\" (UniqueName: \"kubernetes.io/projected/167b1498-09ad-4ab3-84e9-8787235afb26-kube-api-access-mskpj\") pod \"node-resolver-qgk6s\" (UID: \"167b1498-09ad-4ab3-84e9-8787235afb26\") " pod="openshift-dns/node-resolver-qgk6s" Jul 20 17:48:14.249832 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.249805 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9ppkw" Jul 20 17:48:14.277994 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.277963 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87k9h\" (UniqueName: \"kubernetes.io/projected/83a7f11b-2c9a-45ed-923c-4757f07212c9-kube-api-access-87k9h\") pod \"ovnkube-node-m5tpx\" (UID: \"83a7f11b-2c9a-45ed-923c-4757f07212c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.280696 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.280679 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2xsj4" Jul 20 17:48:14.295791 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.295766 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-55xlw" Jul 20 17:48:14.306371 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.306354 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7jrr6" Jul 20 17:48:14.311827 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.311809 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q9g6b" Jul 20 17:48:14.318810 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.318792 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v9j9l" Jul 20 17:48:14.328721 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.328697 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-75t9c" Jul 20 17:48:14.336238 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.336221 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" Jul 20 17:48:14.341997 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.341983 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qgk6s" Jul 20 17:48:14.349163 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.349138 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:14.595610 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.595537 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs\") pod \"network-metrics-daemon-r24gs\" (UID: \"76af6cb3-7b9d-4993-a5c8-24afeb42b60b\") " pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:14.595764 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:14.595644 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:14.595764 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:14.595717 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs podName:76af6cb3-7b9d-4993-a5c8-24afeb42b60b nodeName:}" failed. No retries permitted until 2026-07-20 17:48:15.595703173 +0000 UTC m=+3.967491730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs") pod "network-metrics-daemon-r24gs" (UID: "76af6cb3-7b9d-4993-a5c8-24afeb42b60b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:14.696684 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.696644 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqbk\" (UniqueName: \"kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk\") pod \"network-check-target-8rmfb\" (UID: \"e8d261a8-44f9-4a08-b932-236b2306b4f7\") " pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:14.696842 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:14.696708 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret\") pod \"global-pull-secret-syncer-rshx6\" (UID: \"21af06f0-d59b-4e61-84fb-57425520c8c3\") " pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:14.696842 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:14.696815 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:14.696973 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:14.696850 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 17:48:14.696973 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:14.696875 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret podName:21af06f0-d59b-4e61-84fb-57425520c8c3 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:15.696861721 +0000 UTC m=+4.068650276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret") pod "global-pull-secret-syncer-rshx6" (UID: "21af06f0-d59b-4e61-84fb-57425520c8c3") : object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:14.696973 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:14.696881 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 17:48:14.696973 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:14.696896 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gpqbk for pod openshift-network-diagnostics/network-check-target-8rmfb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:14.696973 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:14.696946 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk podName:e8d261a8-44f9-4a08-b932-236b2306b4f7 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:15.696930109 +0000 UTC m=+4.068718667 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gpqbk" (UniqueName: "kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk") pod "network-check-target-8rmfb" (UID: "e8d261a8-44f9-4a08-b932-236b2306b4f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:14.922645 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:14.922536 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda265dbdd_eaa3_465c_a32b_022b3b94f594.slice/crio-aed787ee2cf23627daf050585b65417fdfaff09d21a8b9c966c1f1ec9bc10547 WatchSource:0}: Error finding container aed787ee2cf23627daf050585b65417fdfaff09d21a8b9c966c1f1ec9bc10547: Status 404 returned error can't find the container with id aed787ee2cf23627daf050585b65417fdfaff09d21a8b9c966c1f1ec9bc10547 Jul 20 17:48:14.924099 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:14.924075 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bf65ad_15fc_46c4_a109_34b8132e2431.slice/crio-92f09cb70de8dd86d564c28192fd20ac12c9c50e1d046aa07216f237b9349f88 WatchSource:0}: Error finding container 92f09cb70de8dd86d564c28192fd20ac12c9c50e1d046aa07216f237b9349f88: Status 404 returned error can't find the container with id 92f09cb70de8dd86d564c28192fd20ac12c9c50e1d046aa07216f237b9349f88 Jul 20 17:48:14.927362 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:14.927338 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167b1498_09ad_4ab3_84e9_8787235afb26.slice/crio-21151cf8dd4a39b64564a10171484697a87dc7b3ab010ab1d98ed99687d7430c WatchSource:0}: Error finding container 21151cf8dd4a39b64564a10171484697a87dc7b3ab010ab1d98ed99687d7430c: Status 404 returned error can't find the container with id 21151cf8dd4a39b64564a10171484697a87dc7b3ab010ab1d98ed99687d7430c Jul 20 17:48:14.928040 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:14.927916 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7194a6f0_d226_438f_8445_2db548350c5b.slice/crio-1452b447a35c186dc1bef22493c833fa27109abf8480d9c92eb6d3a6ff617344 WatchSource:0}: Error finding container 1452b447a35c186dc1bef22493c833fa27109abf8480d9c92eb6d3a6ff617344: Status 404 returned error can't find the container with id 1452b447a35c186dc1bef22493c833fa27109abf8480d9c92eb6d3a6ff617344 Jul 20 17:48:14.928991 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:14.928971 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda819ab4d_1473_490e_831c_733e4d0a729b.slice/crio-00f9056bb335f1564b539aac11913d388149e3910a46d0c8bc8b22fc5f3a4fe7 WatchSource:0}: Error finding container 00f9056bb335f1564b539aac11913d388149e3910a46d0c8bc8b22fc5f3a4fe7: Status 404 returned error can't find the container with id 00f9056bb335f1564b539aac11913d388149e3910a46d0c8bc8b22fc5f3a4fe7 Jul 20 17:48:14.930304 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:14.930280 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8583b5a_bbcd_44ba_9c03_2a14cfa40f19.slice/crio-f4a432de38acedd4faba2e6c72b4615c5d4c6b7ad2dba4a5e71d7cc1c955ca51 WatchSource:0}: Error finding container f4a432de38acedd4faba2e6c72b4615c5d4c6b7ad2dba4a5e71d7cc1c955ca51: Status 404 returned error can't find the container with id f4a432de38acedd4faba2e6c72b4615c5d4c6b7ad2dba4a5e71d7cc1c955ca51 Jul 20 17:48:14.930665 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:14.930645 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff18797_615b_4cc4_82f2_3621080f562e.slice/crio-982c094b2b86b2ba7a1af7e7bf8a88f1dbf490ec6ce93d7ac9d7d1b80df264b8 WatchSource:0}: Error finding container 982c094b2b86b2ba7a1af7e7bf8a88f1dbf490ec6ce93d7ac9d7d1b80df264b8: Status 404 returned error can't find the container with id 982c094b2b86b2ba7a1af7e7bf8a88f1dbf490ec6ce93d7ac9d7d1b80df264b8 Jul 20 17:48:14.932937 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:14.932377 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0df65aa_855d_41d5_9a22_3368606a7700.slice/crio-4f752eea854f01d7bc3eb5ed98026c2d7c8583f30cd58a16a5ab8fc0776aa423 WatchSource:0}: Error finding container 4f752eea854f01d7bc3eb5ed98026c2d7c8583f30cd58a16a5ab8fc0776aa423: Status 404 returned error can't find the container with id 4f752eea854f01d7bc3eb5ed98026c2d7c8583f30cd58a16a5ab8fc0776aa423 Jul 20 17:48:14.933158 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:14.933132 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a7f11b_2c9a_45ed_923c_4757f07212c9.slice/crio-9dfb9cc59176c2ef8cb6d49c332047b0786d6070462d565550debbce3add17bf WatchSource:0}: Error finding container 9dfb9cc59176c2ef8cb6d49c332047b0786d6070462d565550debbce3add17bf: Status 404 returned error can't find the container with id 9dfb9cc59176c2ef8cb6d49c332047b0786d6070462d565550debbce3add17bf Jul 20 17:48:15.164680 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.164474 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:15.164811 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.164479 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:15.164811 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.164483 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:15.164811 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:15.164788 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rmfb" podUID="e8d261a8-44f9-4a08-b932-236b2306b4f7" Jul 20 17:48:15.164944 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:15.164724 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rshx6" podUID="21af06f0-d59b-4e61-84fb-57425520c8c3" Jul 20 17:48:15.164944 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:15.164867 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r24gs" podUID="76af6cb3-7b9d-4993-a5c8-24afeb42b60b" Jul 20 17:48:15.172788 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.172761 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" event={"ID":"83a7f11b-2c9a-45ed-923c-4757f07212c9","Type":"ContainerStarted","Data":"9dfb9cc59176c2ef8cb6d49c332047b0786d6070462d565550debbce3add17bf"} Jul 20 17:48:15.173818 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.173798 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" event={"ID":"a0df65aa-855d-41d5-9a22-3368606a7700","Type":"ContainerStarted","Data":"4f752eea854f01d7bc3eb5ed98026c2d7c8583f30cd58a16a5ab8fc0776aa423"} Jul 20 17:48:15.174789 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.174748 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xsj4" event={"ID":"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19","Type":"ContainerStarted","Data":"f4a432de38acedd4faba2e6c72b4615c5d4c6b7ad2dba4a5e71d7cc1c955ca51"} Jul 20 17:48:15.175937 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.175910 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-55xlw" event={"ID":"a819ab4d-1473-490e-831c-733e4d0a729b","Type":"ContainerStarted","Data":"00f9056bb335f1564b539aac11913d388149e3910a46d0c8bc8b22fc5f3a4fe7"} Jul 20 17:48:15.177034 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.177003 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v9j9l" event={"ID":"7194a6f0-d226-438f-8445-2db548350c5b","Type":"ContainerStarted","Data":"1452b447a35c186dc1bef22493c833fa27109abf8480d9c92eb6d3a6ff617344"} Jul 20 17:48:15.178746 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.178714 2583 generic.go:356] "Generic (PLEG): container finished" podID="1cb3c65fdce6a2649d8a37538aaf404d" containerID="f22898a5b71c5dfd9becf8e6c978fda9c2ce63800cd2bf84f7d1165f1755b8b4" exitCode=0 Jul 20 17:48:15.178832 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.178761 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal" event={"ID":"1cb3c65fdce6a2649d8a37538aaf404d","Type":"ContainerDied","Data":"f22898a5b71c5dfd9becf8e6c978fda9c2ce63800cd2bf84f7d1165f1755b8b4"} Jul 20 17:48:15.179785 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.179764 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7jrr6" event={"ID":"0ff18797-615b-4cc4-82f2-3621080f562e","Type":"ContainerStarted","Data":"982c094b2b86b2ba7a1af7e7bf8a88f1dbf490ec6ce93d7ac9d7d1b80df264b8"} Jul 20 17:48:15.180772 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.180753 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qgk6s" event={"ID":"167b1498-09ad-4ab3-84e9-8787235afb26","Type":"ContainerStarted","Data":"21151cf8dd4a39b64564a10171484697a87dc7b3ab010ab1d98ed99687d7430c"} Jul 20 17:48:15.181706 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.181681 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-75t9c" event={"ID":"f5bf65ad-15fc-46c4-a109-34b8132e2431","Type":"ContainerStarted","Data":"92f09cb70de8dd86d564c28192fd20ac12c9c50e1d046aa07216f237b9349f88"} Jul 20 17:48:15.182661 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.182644 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q9g6b" event={"ID":"a265dbdd-eaa3-465c-a32b-022b3b94f594","Type":"ContainerStarted","Data":"aed787ee2cf23627daf050585b65417fdfaff09d21a8b9c966c1f1ec9bc10547"} Jul 20 17:48:15.185800 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.185783 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-129.ec2.internal" event={"ID":"7240dbae0531c45a7cd9f8a23f8cd206","Type":"ContainerStarted","Data":"3bbc088d611c7d9d216817b39e36f8395caae6f6c6a913104423c4549a87d507"} Jul 20 17:48:15.253103 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.253076 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 17:43:14 +0000 UTC" deadline="2028-03-08 20:56:53.544014293 +0000 UTC" Jul 20 17:48:15.253176 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.253103 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14331h8m38.290914528s" Jul 20 17:48:15.602793 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.602700 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs\") pod \"network-metrics-daemon-r24gs\" (UID: \"76af6cb3-7b9d-4993-a5c8-24afeb42b60b\") " pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:15.602963 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:15.602871 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:15.602963 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:15.602930 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs podName:76af6cb3-7b9d-4993-a5c8-24afeb42b60b nodeName:}" failed. No retries permitted until 2026-07-20 17:48:17.602913916 +0000 UTC m=+5.974702479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs") pod "network-metrics-daemon-r24gs" (UID: "76af6cb3-7b9d-4993-a5c8-24afeb42b60b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:15.703800 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.703721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret\") pod \"global-pull-secret-syncer-rshx6\" (UID: \"21af06f0-d59b-4e61-84fb-57425520c8c3\") " pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:15.703955 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:15.703831 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqbk\" (UniqueName: \"kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk\") pod \"network-check-target-8rmfb\" (UID: \"e8d261a8-44f9-4a08-b932-236b2306b4f7\") " pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:15.704010 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:15.703968 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 17:48:15.704010 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:15.703986 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 17:48:15.704010 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:15.703998 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gpqbk for pod openshift-network-diagnostics/network-check-target-8rmfb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:15.704280 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:15.704055 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk podName:e8d261a8-44f9-4a08-b932-236b2306b4f7 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:17.704037618 +0000 UTC m=+6.075826188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gpqbk" (UniqueName: "kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk") pod "network-check-target-8rmfb" (UID: "e8d261a8-44f9-4a08-b932-236b2306b4f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:15.704280 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:15.704165 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:15.704280 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:15.704217 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret podName:21af06f0-d59b-4e61-84fb-57425520c8c3 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:17.704203964 +0000 UTC m=+6.075992517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret") pod "global-pull-secret-syncer-rshx6" (UID: "21af06f0-d59b-4e61-84fb-57425520c8c3") : object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:16.208833 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:16.208789 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal" event={"ID":"1cb3c65fdce6a2649d8a37538aaf404d","Type":"ContainerStarted","Data":"b7e21a7c45a562f5ecc7c36309dcc8da4a8831de3bd78d0acc39b91ad49e0c91"} Jul 20 17:48:16.253816 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:16.253758 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 17:43:14 +0000 UTC" deadline="2028-01-09 14:14:30.833496416 +0000 UTC" Jul 20 17:48:16.253816 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:16.253814 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12908h26m14.579686479s" Jul 20 17:48:16.290492 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:16.289759 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-129.ec2.internal" podStartSLOduration=3.289719043 podStartE2EDuration="3.289719043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:16.279769151 +0000 UTC m=+4.651557726" watchObservedRunningTime="2026-07-20 17:48:16.289719043 +0000 UTC m=+4.661507619" Jul 20 17:48:16.290492 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:16.290063 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-129.ec2.internal" podStartSLOduration=3.290052086 podStartE2EDuration="3.290052086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:15.290008695 +0000 UTC m=+3.661797268" watchObservedRunningTime="2026-07-20 17:48:16.290052086 +0000 UTC m=+4.661840662" Jul 20 17:48:17.164325 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:17.164294 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:17.164514 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:17.164435 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r24gs" podUID="76af6cb3-7b9d-4993-a5c8-24afeb42b60b" Jul 20 17:48:17.164910 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:17.164891 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:17.165013 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:17.164987 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rshx6" podUID="21af06f0-d59b-4e61-84fb-57425520c8c3" Jul 20 17:48:17.165073 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:17.165048 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:17.165123 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:17.165110 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rmfb" podUID="e8d261a8-44f9-4a08-b932-236b2306b4f7" Jul 20 17:48:17.620057 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:17.619963 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs\") pod \"network-metrics-daemon-r24gs\" (UID: \"76af6cb3-7b9d-4993-a5c8-24afeb42b60b\") " pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:17.620494 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:17.620158 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:17.620494 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:17.620224 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs podName:76af6cb3-7b9d-4993-a5c8-24afeb42b60b nodeName:}" failed. No retries permitted until 2026-07-20 17:48:21.620202576 +0000 UTC m=+9.991991149 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs") pod "network-metrics-daemon-r24gs" (UID: "76af6cb3-7b9d-4993-a5c8-24afeb42b60b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:17.721620 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:17.720911 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqbk\" (UniqueName: \"kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk\") pod \"network-check-target-8rmfb\" (UID: \"e8d261a8-44f9-4a08-b932-236b2306b4f7\") " pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:17.721620 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:17.720964 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret\") pod \"global-pull-secret-syncer-rshx6\" (UID: \"21af06f0-d59b-4e61-84fb-57425520c8c3\") " pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:17.721620 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:17.721090 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:17.721620 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:17.721149 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret podName:21af06f0-d59b-4e61-84fb-57425520c8c3 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:21.721131718 +0000 UTC m=+10.092920276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret") pod "global-pull-secret-syncer-rshx6" (UID: "21af06f0-d59b-4e61-84fb-57425520c8c3") : object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:17.721620 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:17.721523 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 17:48:17.721620 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:17.721541 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 17:48:17.721620 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:17.721552 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gpqbk for pod openshift-network-diagnostics/network-check-target-8rmfb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:17.721620 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:17.721594 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk podName:e8d261a8-44f9-4a08-b932-236b2306b4f7 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:21.721580229 +0000 UTC m=+10.093368781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gpqbk" (UniqueName: "kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk") pod "network-check-target-8rmfb" (UID: "e8d261a8-44f9-4a08-b932-236b2306b4f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:19.164839 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:19.164803 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:19.165394 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:19.164939 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rmfb" podUID="e8d261a8-44f9-4a08-b932-236b2306b4f7" Jul 20 17:48:19.165394 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:19.165048 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:19.165394 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:19.165165 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rshx6" podUID="21af06f0-d59b-4e61-84fb-57425520c8c3" Jul 20 17:48:19.165394 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:19.165214 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:19.165394 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:19.165294 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r24gs" podUID="76af6cb3-7b9d-4993-a5c8-24afeb42b60b" Jul 20 17:48:21.164891 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:21.164860 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:21.165354 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:21.164896 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:21.165354 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:21.164896 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:21.165354 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:21.164991 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rshx6" podUID="21af06f0-d59b-4e61-84fb-57425520c8c3" Jul 20 17:48:21.165514 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:21.165420 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rmfb" podUID="e8d261a8-44f9-4a08-b932-236b2306b4f7" Jul 20 17:48:21.168340 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:21.166269 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r24gs" podUID="76af6cb3-7b9d-4993-a5c8-24afeb42b60b" Jul 20 17:48:21.653542 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:21.653460 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs\") pod \"network-metrics-daemon-r24gs\" (UID: \"76af6cb3-7b9d-4993-a5c8-24afeb42b60b\") " pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:21.653712 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:21.653677 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:21.653820 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:21.653784 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs podName:76af6cb3-7b9d-4993-a5c8-24afeb42b60b nodeName:}" failed. No retries permitted until 2026-07-20 17:48:29.653762525 +0000 UTC m=+18.025551096 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs") pod "network-metrics-daemon-r24gs" (UID: "76af6cb3-7b9d-4993-a5c8-24afeb42b60b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:21.754386 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:21.754350 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqbk\" (UniqueName: \"kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk\") pod \"network-check-target-8rmfb\" (UID: \"e8d261a8-44f9-4a08-b932-236b2306b4f7\") " pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:21.754546 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:21.754480 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret\") pod \"global-pull-secret-syncer-rshx6\" (UID: \"21af06f0-d59b-4e61-84fb-57425520c8c3\") " pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:21.754546 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:21.754520 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 17:48:21.754546 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:21.754544 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 17:48:21.754744 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:21.754559 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gpqbk for pod openshift-network-diagnostics/network-check-target-8rmfb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:21.754744 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:21.754587 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:21.754744 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:21.754623 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk podName:e8d261a8-44f9-4a08-b932-236b2306b4f7 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:29.754605075 +0000 UTC m=+18.126393640 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gpqbk" (UniqueName: "kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk") pod "network-check-target-8rmfb" (UID: "e8d261a8-44f9-4a08-b932-236b2306b4f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:21.754744 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:21.754641 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret podName:21af06f0-d59b-4e61-84fb-57425520c8c3 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:29.754633753 +0000 UTC m=+18.126422308 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret") pod "global-pull-secret-syncer-rshx6" (UID: "21af06f0-d59b-4e61-84fb-57425520c8c3") : object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:23.164895 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:23.164847 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:23.164895 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:23.164890 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:23.165436 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:23.164853 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:23.165436 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:23.164976 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rshx6" podUID="21af06f0-d59b-4e61-84fb-57425520c8c3" Jul 20 17:48:23.165436 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:23.165085 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r24gs" podUID="76af6cb3-7b9d-4993-a5c8-24afeb42b60b" Jul 20 17:48:23.165436 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:23.165173 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rmfb" podUID="e8d261a8-44f9-4a08-b932-236b2306b4f7" Jul 20 17:48:25.165183 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:25.165143 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:25.165618 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:25.165194 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:25.165618 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:25.165194 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:25.165618 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:25.165269 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rmfb" podUID="e8d261a8-44f9-4a08-b932-236b2306b4f7" Jul 20 17:48:25.165618 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:25.165397 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rshx6" podUID="21af06f0-d59b-4e61-84fb-57425520c8c3" Jul 20 17:48:25.165618 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:25.165487 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r24gs" podUID="76af6cb3-7b9d-4993-a5c8-24afeb42b60b" Jul 20 17:48:27.165192 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:27.165154 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:27.165192 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:27.165184 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:27.165747 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:27.165172 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:27.165747 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:27.165300 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r24gs" podUID="76af6cb3-7b9d-4993-a5c8-24afeb42b60b" Jul 20 17:48:27.165747 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:27.165396 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rshx6" podUID="21af06f0-d59b-4e61-84fb-57425520c8c3" Jul 20 17:48:27.165747 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:27.165476 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rmfb" podUID="e8d261a8-44f9-4a08-b932-236b2306b4f7" Jul 20 17:48:29.164250 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:29.164212 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:29.164816 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:29.164311 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:29.164816 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:29.164323 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rmfb" podUID="e8d261a8-44f9-4a08-b932-236b2306b4f7" Jul 20 17:48:29.164816 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:29.164347 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:29.164816 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:29.164459 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rshx6" podUID="21af06f0-d59b-4e61-84fb-57425520c8c3" Jul 20 17:48:29.164816 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:29.164555 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r24gs" podUID="76af6cb3-7b9d-4993-a5c8-24afeb42b60b" Jul 20 17:48:29.714991 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:29.714949 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs\") pod \"network-metrics-daemon-r24gs\" (UID: \"76af6cb3-7b9d-4993-a5c8-24afeb42b60b\") " pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:29.715186 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:29.715117 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:29.715251 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:29.715195 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs podName:76af6cb3-7b9d-4993-a5c8-24afeb42b60b nodeName:}" failed. No retries permitted until 2026-07-20 17:48:45.715173892 +0000 UTC m=+34.086962461 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs") pod "network-metrics-daemon-r24gs" (UID: "76af6cb3-7b9d-4993-a5c8-24afeb42b60b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:29.816293 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:29.816254 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqbk\" (UniqueName: \"kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk\") pod \"network-check-target-8rmfb\" (UID: \"e8d261a8-44f9-4a08-b932-236b2306b4f7\") " pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:29.816465 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:29.816316 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret\") pod \"global-pull-secret-syncer-rshx6\" (UID: \"21af06f0-d59b-4e61-84fb-57425520c8c3\") " pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:29.816465 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:29.816445 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:29.816465 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:29.816450 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 17:48:29.816634 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:29.816481 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 17:48:29.816634 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:29.816496 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gpqbk for pod openshift-network-diagnostics/network-check-target-8rmfb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:29.816634 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:29.816500 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret podName:21af06f0-d59b-4e61-84fb-57425520c8c3 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:45.816487329 +0000 UTC m=+34.188275880 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret") pod "global-pull-secret-syncer-rshx6" (UID: "21af06f0-d59b-4e61-84fb-57425520c8c3") : object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:29.816634 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:29.816541 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk podName:e8d261a8-44f9-4a08-b932-236b2306b4f7 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:45.816529536 +0000 UTC m=+34.188318091 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gpqbk" (UniqueName: "kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk") pod "network-check-target-8rmfb" (UID: "e8d261a8-44f9-4a08-b932-236b2306b4f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:31.164653 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:31.164583 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:31.164653 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:31.164610 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:31.165214 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:31.164705 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:31.165214 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:31.164704 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rmfb" podUID="e8d261a8-44f9-4a08-b932-236b2306b4f7" Jul 20 17:48:31.165214 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:31.164864 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r24gs" podUID="76af6cb3-7b9d-4993-a5c8-24afeb42b60b" Jul 20 17:48:31.165214 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:31.164937 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rshx6" podUID="21af06f0-d59b-4e61-84fb-57425520c8c3" Jul 20 17:48:33.164503 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:33.164469 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:33.164908 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:33.164589 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:33.164908 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:33.164594 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:33.164908 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:33.164589 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rmfb" podUID="e8d261a8-44f9-4a08-b932-236b2306b4f7" Jul 20 17:48:33.164908 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:33.164720 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r24gs" podUID="76af6cb3-7b9d-4993-a5c8-24afeb42b60b" Jul 20 17:48:33.164908 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:33.164821 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rshx6" podUID="21af06f0-d59b-4e61-84fb-57425520c8c3" Jul 20 17:48:34.253029 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:34.252669 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7jrr6" event={"ID":"0ff18797-615b-4cc4-82f2-3621080f562e","Type":"ContainerStarted","Data":"8b5cdf044b8ea810d5b483c02aa37e52bc852a5fcc4f2781b7b9474d7695f5d2"} Jul 20 17:48:34.254052 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:34.254030 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qgk6s" event={"ID":"167b1498-09ad-4ab3-84e9-8787235afb26","Type":"ContainerStarted","Data":"c2577600bca7aa833d4cdb012c0e4e754bfb74682f4b3ce94fa80aa283cc63e5"} Jul 20 17:48:34.255211 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:34.255183 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q9g6b" event={"ID":"a265dbdd-eaa3-465c-a32b-022b3b94f594","Type":"ContainerStarted","Data":"32e466396f7774a097a2fca6a962220e3258bd9079f0ab0a381523d52a24f2d6"} Jul 20 17:48:34.257348 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:34.257328 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" event={"ID":"83a7f11b-2c9a-45ed-923c-4757f07212c9","Type":"ContainerStarted","Data":"669c822a307f98fc1bd07cd3c27518e9402c836192c58cec1951f8fde334201a"} Jul 20 17:48:34.257434 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:34.257398 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" event={"ID":"83a7f11b-2c9a-45ed-923c-4757f07212c9","Type":"ContainerStarted","Data":"048adc9a52065df3385da8818c96c68ab90619fa421641596ce9815a235b8543"} Jul 20 17:48:34.257481 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:34.257445 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" event={"ID":"83a7f11b-2c9a-45ed-923c-4757f07212c9","Type":"ContainerStarted","Data":"836548a411d86114808910f670ba3766f1162a6498d3c14eedbe0be45795238d"} Jul 20 17:48:34.257515 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:34.257490 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" event={"ID":"83a7f11b-2c9a-45ed-923c-4757f07212c9","Type":"ContainerStarted","Data":"8c2e8548154fad6f6021bb64bb1ac0fb8db4e6e846e5ff0682b48dc56f951313"} Jul 20 17:48:34.258637 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:34.258618 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" event={"ID":"a0df65aa-855d-41d5-9a22-3368606a7700","Type":"ContainerStarted","Data":"5b43bec7650b27521ea9fca8e5b335b038251f7ad9c79e3351a3c60f1ddee675"} Jul 20 17:48:34.259855 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:34.259833 2583 generic.go:356] "Generic (PLEG): container finished" podID="e8583b5a-bbcd-44ba-9c03-2a14cfa40f19" containerID="b1a56243fc7bd0e76c6ee7e37f85ef9f687c777287e05f2403aa7db75acdf70c" exitCode=0 Jul 20 17:48:34.259935 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:34.259904 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xsj4" event={"ID":"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19","Type":"ContainerDied","Data":"b1a56243fc7bd0e76c6ee7e37f85ef9f687c777287e05f2403aa7db75acdf70c"} Jul 20 17:48:34.261218 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:34.261190 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-55xlw" event={"ID":"a819ab4d-1473-490e-831c-733e4d0a729b","Type":"ContainerStarted","Data":"8683756a9e921161a1385172b7c360e8e0b22d62c61c7f8a0b7f2ff245e6c3c1"} Jul 20 17:48:34.262488 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:34.262469 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v9j9l" event={"ID":"7194a6f0-d226-438f-8445-2db548350c5b","Type":"ContainerStarted","Data":"0c9d6086f19498609ddb8bccc2a07a8f125ea828bf69a3182e819c9841e3f3c1"} Jul 20 17:48:34.327740 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:34.327667 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-55xlw" podStartSLOduration=3.593532798 podStartE2EDuration="22.327652175s" totalImagesPullingTime="18.734119377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:34.327525318 +0000 UTC m=+22.699313894" watchObservedRunningTime="2026-07-20 17:48:34.327652175 +0000 UTC m=+22.699440748" Jul 20 17:48:34.327868 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:34.327814 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7jrr6" podStartSLOduration=3.596058067 podStartE2EDuration="22.327807579s" totalImagesPullingTime="18.731749512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:34.28002229 +0000 UTC m=+22.651810866" watchObservedRunningTime="2026-07-20 17:48:34.327807579 +0000 UTC m=+22.699596154" Jul 20 17:48:34.403209 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:34.403152 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-q9g6b" podStartSLOduration=3.6097426969999997 podStartE2EDuration="22.403130529s" totalImagesPullingTime="18.793387832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:34.365535214 +0000 UTC m=+22.737323788" watchObservedRunningTime="2026-07-20 17:48:34.403130529 +0000 UTC m=+22.774919102" Jul 20 17:48:34.468703 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:34.468603 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qgk6s" podStartSLOduration=3.734716019 podStartE2EDuration="22.468584084s" totalImagesPullingTime="18.733868065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:34.402991676 +0000 UTC m=+22.774780251" watchObservedRunningTime="2026-07-20 17:48:34.468584084 +0000 UTC m=+22.840372657" Jul 20 17:48:34.496190 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:34.496147 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v9j9l" podStartSLOduration=8.235128981999999 podStartE2EDuration="22.496130119s" totalImagesPullingTime="14.261001137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:34.495675689 +0000 UTC m=+22.867464277" watchObservedRunningTime="2026-07-20 17:48:34.496130119 +0000 UTC m=+22.867918692" Jul 20 17:48:35.091069 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:35.091036 2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 20 17:48:35.151215 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:35.151184 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v9j9l" Jul 20 17:48:35.151837 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:35.151817 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v9j9l" Jul 20 17:48:35.164765 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:35.164744 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:35.164909 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:35.164823 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:35.164909 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:35.164827 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rmfb" podUID="e8d261a8-44f9-4a08-b932-236b2306b4f7" Jul 20 17:48:35.164909 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:35.164849 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:35.165077 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:35.164925 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r24gs" podUID="76af6cb3-7b9d-4993-a5c8-24afeb42b60b" Jul 20 17:48:35.165077 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:35.164987 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rshx6" podUID="21af06f0-d59b-4e61-84fb-57425520c8c3" Jul 20 17:48:35.266549 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:35.266515 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-75t9c" event={"ID":"f5bf65ad-15fc-46c4-a109-34b8132e2431","Type":"ContainerStarted","Data":"3a534459eccf403f135e57360ff89c560c8b2a475f98f0c63c04e06d6e0d48fc"} Jul 20 17:48:35.269358 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:35.269330 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" event={"ID":"83a7f11b-2c9a-45ed-923c-4757f07212c9","Type":"ContainerStarted","Data":"8b043cbb85ed0fa23543c6237450650722b50fd7506e1fc4560a5a10aaf05be9"} Jul 20 17:48:35.269479 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:35.269400 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" event={"ID":"83a7f11b-2c9a-45ed-923c-4757f07212c9","Type":"ContainerStarted","Data":"4503af84c2e688130e6bcaa23b618cfd6b7812e3e2c5701473b14ae215d86c36"} Jul 20 17:48:35.270899 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:35.270878 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" event={"ID":"a0df65aa-855d-41d5-9a22-3368606a7700","Type":"ContainerStarted","Data":"9702f8baf5250cce0c727e003cd1320434abd070c3f53d28796c842902a863f2"} Jul 20 17:48:35.271345 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:35.271321 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v9j9l" Jul 20 17:48:35.271785 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:35.271769 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v9j9l" Jul 20 17:48:35.318626 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:35.318589 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-75t9c" podStartSLOduration=4.580852895 podStartE2EDuration="23.318577554s" totalImagesPullingTime="18.737724659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:35.316913575 +0000 UTC m=+23.688702146" watchObservedRunningTime="2026-07-20 17:48:35.318577554 +0000 UTC m=+23.690366129" Jul 20 17:48:36.043794 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:36.043669 2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T17:48:35.091057446Z","UUID":"6cec6f2e-b22c-4193-8de4-d577a3531edb","Handler":null,"Name":"","Endpoint":""} Jul 20 17:48:36.045346 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:36.045321 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 20 17:48:36.045467 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:36.045366 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 20 17:48:37.165034 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:37.164994 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:37.165506 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:37.165042 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:37.165506 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:37.165012 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:37.165506 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:37.165140 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r24gs" podUID="76af6cb3-7b9d-4993-a5c8-24afeb42b60b" Jul 20 17:48:37.165506 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:37.165213 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rmfb" podUID="e8d261a8-44f9-4a08-b932-236b2306b4f7" Jul 20 17:48:37.165506 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:37.165277 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rshx6" podUID="21af06f0-d59b-4e61-84fb-57425520c8c3" Jul 20 17:48:37.279366 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:37.279333 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" event={"ID":"83a7f11b-2c9a-45ed-923c-4757f07212c9","Type":"ContainerStarted","Data":"69dd465ea4a4933f8450ff481f7294f6c3c9c7d7e4ee060fe960e332a7c12fa4"} Jul 20 17:48:37.281795 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:37.281762 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" event={"ID":"a0df65aa-855d-41d5-9a22-3368606a7700","Type":"ContainerStarted","Data":"d9cc0153ceb5a1fcbb6bfb37ce04182cbd98ef8bf0e9d3a5ea30990d5e4f410c"} Jul 20 17:48:37.316703 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:37.316660 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567wk" podStartSLOduration=4.251378663 podStartE2EDuration="25.316647153s" totalImagesPullingTime="21.06526849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:48:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:37.316515507 +0000 UTC m=+25.688304094" watchObservedRunningTime="2026-07-20 17:48:37.316647153 +0000 UTC m=+25.688435726" Jul 20 17:48:39.165170 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:39.164997 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:39.165604 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:39.165022 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:39.165604 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:39.165246 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rshx6" podUID="21af06f0-d59b-4e61-84fb-57425520c8c3" Jul 20 17:48:39.165604 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:39.165358 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r24gs" podUID="76af6cb3-7b9d-4993-a5c8-24afeb42b60b" Jul 20 17:48:39.165604 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:39.165023 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:39.165604 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:39.165451 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rmfb" podUID="e8d261a8-44f9-4a08-b932-236b2306b4f7" Jul 20 17:48:39.290448 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:39.290411 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" event={"ID":"83a7f11b-2c9a-45ed-923c-4757f07212c9","Type":"ContainerStarted","Data":"75ac10bc4a80d55b67c40daaf6cbabcf9aca1a48aa80ed3b923f681474a73c83"} Jul 20 17:48:40.294088 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:40.293900 2583 generic.go:356] "Generic (PLEG): container finished" podID="e8583b5a-bbcd-44ba-9c03-2a14cfa40f19" containerID="afb71087411eabc94e5ae126c241521e8e41a75d0a4364560abcc37d1a881052" exitCode=0 Jul 20 17:48:40.294763 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:40.293989 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xsj4" event={"ID":"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19","Type":"ContainerDied","Data":"afb71087411eabc94e5ae126c241521e8e41a75d0a4364560abcc37d1a881052"} Jul 20 17:48:40.294763 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:40.294363 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:40.310588 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:40.310565 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:40.369812 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:40.369766 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" podStartSLOduration=9.564907745 podStartE2EDuration="28.369751515s" totalImagesPullingTime="18.80484377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:40.36949077 +0000 UTC m=+28.741279344" watchObservedRunningTime="2026-07-20 17:48:40.369751515 +0000 UTC m=+28.741540119" Jul 20 17:48:41.164765 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:41.164714 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:41.164965 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:41.164773 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:41.164965 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:41.164773 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:41.164965 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:41.164836 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rshx6" podUID="21af06f0-d59b-4e61-84fb-57425520c8c3" Jul 20 17:48:41.165861 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:41.165267 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r24gs" podUID="76af6cb3-7b9d-4993-a5c8-24afeb42b60b" Jul 20 17:48:41.165861 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:41.165375 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rmfb" podUID="e8d261a8-44f9-4a08-b932-236b2306b4f7" Jul 20 17:48:41.296789 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:41.296758 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:41.297194 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:41.296857 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:41.310763 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:41.310719 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:48:41.603157 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:41.603084 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rshx6"] Jul 20 17:48:41.603333 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:41.603213 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:41.603333 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:41.603315 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rshx6" podUID="21af06f0-d59b-4e61-84fb-57425520c8c3" Jul 20 17:48:41.605715 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:41.605692 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8rmfb"] Jul 20 17:48:41.605872 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:41.605812 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:41.605943 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:41.605913 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rmfb" podUID="e8d261a8-44f9-4a08-b932-236b2306b4f7" Jul 20 17:48:41.606302 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:41.606280 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r24gs"] Jul 20 17:48:41.606389 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:41.606367 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:41.606458 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:41.606444 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r24gs" podUID="76af6cb3-7b9d-4993-a5c8-24afeb42b60b" Jul 20 17:48:42.300314 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:42.300283 2583 generic.go:356] "Generic (PLEG): container finished" podID="e8583b5a-bbcd-44ba-9c03-2a14cfa40f19" containerID="eb74ca804761da1638bc9202a1f7144039e55d7e3f954bbdf32c96b9fd60b23a" exitCode=0 Jul 20 17:48:42.300720 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:42.300368 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xsj4" event={"ID":"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19","Type":"ContainerDied","Data":"eb74ca804761da1638bc9202a1f7144039e55d7e3f954bbdf32c96b9fd60b23a"} Jul 20 17:48:43.164505 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:43.164331 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:43.164613 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:43.164358 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:43.164613 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:43.164602 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rshx6" podUID="21af06f0-d59b-4e61-84fb-57425520c8c3" Jul 20 17:48:43.164692 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:43.164641 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rmfb" podUID="e8d261a8-44f9-4a08-b932-236b2306b4f7" Jul 20 17:48:43.164692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:43.164365 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:43.164784 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:43.164744 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r24gs" podUID="76af6cb3-7b9d-4993-a5c8-24afeb42b60b" Jul 20 17:48:43.304926 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:43.304892 2583 generic.go:356] "Generic (PLEG): container finished" podID="e8583b5a-bbcd-44ba-9c03-2a14cfa40f19" containerID="c7112837d649a640133c47381a115bc9ac16082d6920862fb01e58c5f8abc0b1" exitCode=0 Jul 20 17:48:43.305241 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:43.304930 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xsj4" event={"ID":"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19","Type":"ContainerDied","Data":"c7112837d649a640133c47381a115bc9ac16082d6920862fb01e58c5f8abc0b1"} Jul 20 17:48:45.164441 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:45.164367 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:45.165008 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:45.164367 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:45.165008 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:45.164497 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r24gs" podUID="76af6cb3-7b9d-4993-a5c8-24afeb42b60b" Jul 20 17:48:45.165008 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:45.164367 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:45.165008 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:45.164570 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rmfb" podUID="e8d261a8-44f9-4a08-b932-236b2306b4f7" Jul 20 17:48:45.165008 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:45.164650 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rshx6" podUID="21af06f0-d59b-4e61-84fb-57425520c8c3" Jul 20 17:48:45.733132 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:45.733095 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs\") pod \"network-metrics-daemon-r24gs\" (UID: \"76af6cb3-7b9d-4993-a5c8-24afeb42b60b\") " pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:45.733292 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:45.733236 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:45.733332 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:45.733296 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs podName:76af6cb3-7b9d-4993-a5c8-24afeb42b60b nodeName:}" failed. No retries permitted until 2026-07-20 17:49:17.733280925 +0000 UTC m=+66.105069480 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs") pod "network-metrics-daemon-r24gs" (UID: "76af6cb3-7b9d-4993-a5c8-24afeb42b60b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:45.834210 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:45.834166 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret\") pod \"global-pull-secret-syncer-rshx6\" (UID: \"21af06f0-d59b-4e61-84fb-57425520c8c3\") " pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:45.834377 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:45.834244 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqbk\" (UniqueName: \"kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk\") pod \"network-check-target-8rmfb\" (UID: \"e8d261a8-44f9-4a08-b932-236b2306b4f7\") " pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:45.834377 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:45.834337 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:45.834377 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:45.834370 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 17:48:45.834527 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:45.834391 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 17:48:45.834527 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:45.834404 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gpqbk for pod openshift-network-diagnostics/network-check-target-8rmfb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:45.834527 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:45.834411 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret podName:21af06f0-d59b-4e61-84fb-57425520c8c3 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:17.834390954 +0000 UTC m=+66.206179522 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret") pod "global-pull-secret-syncer-rshx6" (UID: "21af06f0-d59b-4e61-84fb-57425520c8c3") : object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:45.834527 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:45.834453 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk podName:e8d261a8-44f9-4a08-b932-236b2306b4f7 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:17.834437565 +0000 UTC m=+66.206226122 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gpqbk" (UniqueName: "kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk") pod "network-check-target-8rmfb" (UID: "e8d261a8-44f9-4a08-b932-236b2306b4f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:46.481933 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.481908 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-129.ec2.internal" event="NodeReady" Jul 20 17:48:46.482297 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.482015 2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 20 17:48:46.565928 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.565897 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qvw7j"] Jul 20 17:48:46.567986 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.567954 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qvw7j" Jul 20 17:48:46.573086 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.573065 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 20 17:48:46.573214 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.573109 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nkvwc\"" Jul 20 17:48:46.573214 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.573179 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 20 17:48:46.574719 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.574701 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9rqgb"] Jul 20 17:48:46.576455 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.576439 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9rqgb" Jul 20 17:48:46.580096 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.580078 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 20 17:48:46.580485 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.580467 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 20 17:48:46.580579 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.580475 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-v69jf\"" Jul 20 17:48:46.581271 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.581251 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 20 17:48:46.598494 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.598468 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9rqgb"] Jul 20 17:48:46.600605 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.600583 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qvw7j"] Jul 20 17:48:46.639874 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.639849 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:48:46.640034 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.639881 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg2d9\" (UniqueName: \"kubernetes.io/projected/9477e1ed-8e61-4f89-8cc3-db20326fce10-kube-api-access-bg2d9\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:48:46.640034 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.639925 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9477e1ed-8e61-4f89-8cc3-db20326fce10-tmp-dir\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:48:46.640034 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.639966 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9477e1ed-8e61-4f89-8cc3-db20326fce10-config-volume\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:48:46.740381 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.740304 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9477e1ed-8e61-4f89-8cc3-db20326fce10-config-volume\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:48:46.740548 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.740389 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c931ca0-fe6b-4e41-8bf8-53617767759f-cert\") pod \"ingress-canary-9rqgb\" (UID: \"5c931ca0-fe6b-4e41-8bf8-53617767759f\") " pod="openshift-ingress-canary/ingress-canary-9rqgb" Jul 20 17:48:46.740548 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.740432 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:48:46.740548 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.740460 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg2d9\" (UniqueName: \"kubernetes.io/projected/9477e1ed-8e61-4f89-8cc3-db20326fce10-kube-api-access-bg2d9\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:48:46.740708 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:46.740562 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 17:48:46.740708 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.740568 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srwbn\" (UniqueName: \"kubernetes.io/projected/5c931ca0-fe6b-4e41-8bf8-53617767759f-kube-api-access-srwbn\") pod \"ingress-canary-9rqgb\" (UID: \"5c931ca0-fe6b-4e41-8bf8-53617767759f\") " pod="openshift-ingress-canary/ingress-canary-9rqgb" Jul 20 17:48:46.740708 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:46.740627 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls podName:9477e1ed-8e61-4f89-8cc3-db20326fce10 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:47.240607899 +0000 UTC m=+35.612396462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls") pod "dns-default-qvw7j" (UID: "9477e1ed-8e61-4f89-8cc3-db20326fce10") : secret "dns-default-metrics-tls" not found Jul 20 17:48:46.740893 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.740705 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9477e1ed-8e61-4f89-8cc3-db20326fce10-tmp-dir\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:48:46.740988 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.740969 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9477e1ed-8e61-4f89-8cc3-db20326fce10-tmp-dir\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:48:46.741042 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.740970 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9477e1ed-8e61-4f89-8cc3-db20326fce10-config-volume\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:48:46.762668 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.762632 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg2d9\" (UniqueName: \"kubernetes.io/projected/9477e1ed-8e61-4f89-8cc3-db20326fce10-kube-api-access-bg2d9\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:48:46.842015 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.841981 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c931ca0-fe6b-4e41-8bf8-53617767759f-cert\") pod \"ingress-canary-9rqgb\" (UID: \"5c931ca0-fe6b-4e41-8bf8-53617767759f\") " pod="openshift-ingress-canary/ingress-canary-9rqgb" Jul 20 17:48:46.842182 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.842032 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srwbn\" (UniqueName: \"kubernetes.io/projected/5c931ca0-fe6b-4e41-8bf8-53617767759f-kube-api-access-srwbn\") pod \"ingress-canary-9rqgb\" (UID: \"5c931ca0-fe6b-4e41-8bf8-53617767759f\") " pod="openshift-ingress-canary/ingress-canary-9rqgb" Jul 20 17:48:46.844780 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.844747 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c931ca0-fe6b-4e41-8bf8-53617767759f-cert\") pod \"ingress-canary-9rqgb\" (UID: \"5c931ca0-fe6b-4e41-8bf8-53617767759f\") " pod="openshift-ingress-canary/ingress-canary-9rqgb" Jul 20 17:48:46.851958 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.851932 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srwbn\" (UniqueName: \"kubernetes.io/projected/5c931ca0-fe6b-4e41-8bf8-53617767759f-kube-api-access-srwbn\") pod \"ingress-canary-9rqgb\" (UID: \"5c931ca0-fe6b-4e41-8bf8-53617767759f\") " pod="openshift-ingress-canary/ingress-canary-9rqgb" Jul 20 17:48:46.886885 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:46.886848 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9rqgb" Jul 20 17:48:47.164592 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:47.164510 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:48:47.164592 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:47.164548 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:48:47.164592 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:47.164548 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:48:47.169211 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:47.169191 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 17:48:47.169337 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:47.169199 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mjfj5\"" Jul 20 17:48:47.170035 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:47.170016 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 17:48:47.171898 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:47.171878 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zcx42\"" Jul 20 17:48:47.174745 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:47.174714 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 17:48:47.174836 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:47.174795 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 17:48:47.245579 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:47.245528 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:48:47.245780 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:47.245762 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 17:48:47.245882 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:47.245868 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls podName:9477e1ed-8e61-4f89-8cc3-db20326fce10 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:48.245846593 +0000 UTC m=+36.617635159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls") pod "dns-default-qvw7j" (UID: "9477e1ed-8e61-4f89-8cc3-db20326fce10") : secret "dns-default-metrics-tls" not found Jul 20 17:48:48.252939 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:48.252901 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:48:48.253681 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:48.253125 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 17:48:48.253681 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:48.253206 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls podName:9477e1ed-8e61-4f89-8cc3-db20326fce10 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:50.253186082 +0000 UTC m=+38.624974649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls") pod "dns-default-qvw7j" (UID: "9477e1ed-8e61-4f89-8cc3-db20326fce10") : secret "dns-default-metrics-tls" not found Jul 20 17:48:48.359448 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:48.359257 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9rqgb"] Jul 20 17:48:48.363019 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:48:48.362992 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c931ca0_fe6b_4e41_8bf8_53617767759f.slice/crio-17f0df0a507c0704944a34c14100f0f98bf85147e5e7e3707af13371f00b19c8 WatchSource:0}: Error finding container 17f0df0a507c0704944a34c14100f0f98bf85147e5e7e3707af13371f00b19c8: Status 404 returned error can't find the container with id 17f0df0a507c0704944a34c14100f0f98bf85147e5e7e3707af13371f00b19c8 Jul 20 17:48:49.323596 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:49.323563 2583 generic.go:356] "Generic (PLEG): container finished" podID="e8583b5a-bbcd-44ba-9c03-2a14cfa40f19" containerID="336d4dc6a1bc849bb07ceb49592d0d9b769ae28530b5c89178d204d0241caf86" exitCode=0 Jul 20 17:48:49.324095 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:49.323650 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xsj4" event={"ID":"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19","Type":"ContainerDied","Data":"336d4dc6a1bc849bb07ceb49592d0d9b769ae28530b5c89178d204d0241caf86"} Jul 20 17:48:49.325475 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:49.325445 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9rqgb" event={"ID":"5c931ca0-fe6b-4e41-8bf8-53617767759f","Type":"ContainerStarted","Data":"17f0df0a507c0704944a34c14100f0f98bf85147e5e7e3707af13371f00b19c8"} Jul 20 17:48:50.270428 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:50.270392 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:48:50.270608 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:50.270583 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 17:48:50.270680 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:50.270668 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls podName:9477e1ed-8e61-4f89-8cc3-db20326fce10 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:54.270647569 +0000 UTC m=+42.642436128 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls") pod "dns-default-qvw7j" (UID: "9477e1ed-8e61-4f89-8cc3-db20326fce10") : secret "dns-default-metrics-tls" not found Jul 20 17:48:50.337626 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:50.337592 2583 generic.go:356] "Generic (PLEG): container finished" podID="e8583b5a-bbcd-44ba-9c03-2a14cfa40f19" containerID="b6b8cbcf4e87bf05c1df86fde45a958a88e30358e338625669c6a8dc1eed6e32" exitCode=0 Jul 20 17:48:50.338146 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:50.337654 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xsj4" event={"ID":"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19","Type":"ContainerDied","Data":"b6b8cbcf4e87bf05c1df86fde45a958a88e30358e338625669c6a8dc1eed6e32"} Jul 20 17:48:51.346374 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:51.346191 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xsj4" event={"ID":"e8583b5a-bbcd-44ba-9c03-2a14cfa40f19","Type":"ContainerStarted","Data":"c4ca88df2c9cb084c422cb0353ae78a5a97ac9825bb65e3ad6845f697ac9b368"} Jul 20 17:48:51.347468 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:51.347443 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9rqgb" event={"ID":"5c931ca0-fe6b-4e41-8bf8-53617767759f","Type":"ContainerStarted","Data":"d46789a57c5a9e364bb6b53dc35f26da54f7ae65591bbbfe8c1c06c953e9b1f8"} Jul 20 17:48:51.396818 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:51.396772 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9rqgb" podStartSLOduration=2.863306577 podStartE2EDuration="5.39675643s" totalImagesPullingTime="2.533449853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:51.396522858 +0000 UTC m=+39.768311432" watchObservedRunningTime="2026-07-20 17:48:51.39675643 +0000 UTC m=+39.768545003" Jul 20 17:48:51.396991 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:51.396973 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2xsj4" podStartSLOduration=8.879811137 podStartE2EDuration="39.396969917s" totalImagesPullingTime="30.51715878s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:48:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:51.374537764 +0000 UTC m=+39.746326348" watchObservedRunningTime="2026-07-20 17:48:51.396969917 +0000 UTC m=+39.768758491" Jul 20 17:48:54.300280 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:48:54.300241 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:48:54.300799 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:54.300362 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 17:48:54.300799 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:48:54.300413 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls podName:9477e1ed-8e61-4f89-8cc3-db20326fce10 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:02.300399461 +0000 UTC m=+50.672188014 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls") pod "dns-default-qvw7j" (UID: "9477e1ed-8e61-4f89-8cc3-db20326fce10") : secret "dns-default-metrics-tls" not found Jul 20 17:49:02.357397 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:02.357354 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:49:02.357895 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:02.357539 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 17:49:02.357895 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:02.357601 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls podName:9477e1ed-8e61-4f89-8cc3-db20326fce10 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:18.357582466 +0000 UTC m=+66.729371022 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls") pod "dns-default-qvw7j" (UID: "9477e1ed-8e61-4f89-8cc3-db20326fce10") : secret "dns-default-metrics-tls" not found Jul 20 17:49:13.315621 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:13.315597 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m5tpx" Jul 20 17:49:17.755624 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:17.755584 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs\") pod \"network-metrics-daemon-r24gs\" (UID: \"76af6cb3-7b9d-4993-a5c8-24afeb42b60b\") " pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:49:17.758496 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:17.758479 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 17:49:17.766247 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:17.766232 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 17:49:17.766303 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:17.766284 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs podName:76af6cb3-7b9d-4993-a5c8-24afeb42b60b nodeName:}" failed. No retries permitted until 2026-07-20 17:50:21.76627033 +0000 UTC m=+130.138058882 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs") pod "network-metrics-daemon-r24gs" (UID: "76af6cb3-7b9d-4993-a5c8-24afeb42b60b") : secret "metrics-daemon-secret" not found Jul 20 17:49:17.856791 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:17.856755 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqbk\" (UniqueName: \"kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk\") pod \"network-check-target-8rmfb\" (UID: \"e8d261a8-44f9-4a08-b932-236b2306b4f7\") " pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:49:17.856944 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:17.856806 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret\") pod \"global-pull-secret-syncer-rshx6\" (UID: \"21af06f0-d59b-4e61-84fb-57425520c8c3\") " pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:49:17.860418 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:17.860398 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 17:49:17.860530 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:17.860417 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 17:49:17.869783 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:17.869764 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 17:49:17.871047 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:17.871028 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21af06f0-d59b-4e61-84fb-57425520c8c3-original-pull-secret\") pod \"global-pull-secret-syncer-rshx6\" (UID: \"21af06f0-d59b-4e61-84fb-57425520c8c3\") " pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:49:17.880204 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:17.880180 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpqbk\" (UniqueName: \"kubernetes.io/projected/e8d261a8-44f9-4a08-b932-236b2306b4f7-kube-api-access-gpqbk\") pod \"network-check-target-8rmfb\" (UID: \"e8d261a8-44f9-4a08-b932-236b2306b4f7\") " pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:49:18.079404 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:18.079328 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zcx42\"" Jul 20 17:49:18.083692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:18.083676 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rshx6" Jul 20 17:49:18.086772 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:18.086752 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:49:18.242352 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:18.242257 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8rmfb"] Jul 20 17:49:18.244899 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:49:18.244868 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d261a8_44f9_4a08_b932_236b2306b4f7.slice/crio-62879c4955a364c1eb286cbfe5ffe3220023f0c45c1f5845be2d240b7ba87a59 WatchSource:0}: Error finding container 62879c4955a364c1eb286cbfe5ffe3220023f0c45c1f5845be2d240b7ba87a59: Status 404 returned error can't find the container with id 62879c4955a364c1eb286cbfe5ffe3220023f0c45c1f5845be2d240b7ba87a59 Jul 20 17:49:18.254491 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:18.254465 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rshx6"] Jul 20 17:49:18.257564 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:49:18.257543 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21af06f0_d59b_4e61_84fb_57425520c8c3.slice/crio-b3dd918999c70fe81d0bad8e849aa429cdc0e0277e158cdd7e683b8b5d04ef62 WatchSource:0}: Error finding container b3dd918999c70fe81d0bad8e849aa429cdc0e0277e158cdd7e683b8b5d04ef62: Status 404 returned error can't find the container with id b3dd918999c70fe81d0bad8e849aa429cdc0e0277e158cdd7e683b8b5d04ef62 Jul 20 17:49:18.360639 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:18.360541 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:49:18.360800 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:18.360705 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 17:49:18.360800 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:18.360789 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls podName:9477e1ed-8e61-4f89-8cc3-db20326fce10 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:50.360773572 +0000 UTC m=+98.732562129 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls") pod "dns-default-qvw7j" (UID: "9477e1ed-8e61-4f89-8cc3-db20326fce10") : secret "dns-default-metrics-tls" not found Jul 20 17:49:18.411976 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:18.411938 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8rmfb" event={"ID":"e8d261a8-44f9-4a08-b932-236b2306b4f7","Type":"ContainerStarted","Data":"62879c4955a364c1eb286cbfe5ffe3220023f0c45c1f5845be2d240b7ba87a59"} Jul 20 17:49:18.412925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:18.412905 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rshx6" event={"ID":"21af06f0-d59b-4e61-84fb-57425520c8c3","Type":"ContainerStarted","Data":"b3dd918999c70fe81d0bad8e849aa429cdc0e0277e158cdd7e683b8b5d04ef62"} Jul 20 17:49:23.427969 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:23.427931 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8rmfb" event={"ID":"e8d261a8-44f9-4a08-b932-236b2306b4f7","Type":"ContainerStarted","Data":"ac3a41887996fa9b0c0c4363cb696bb857a0d0cdc2c2beae7637dcf3f927a4e3"} Jul 20 17:49:23.428420 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:23.428024 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:49:23.429450 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:23.429422 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rshx6" event={"ID":"21af06f0-d59b-4e61-84fb-57425520c8c3","Type":"ContainerStarted","Data":"bafb117b1c5b6ca1c1f26b5337458d92a14d43ab5488845c182087c169d6f773"} Jul 20 17:49:23.449352 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:23.449301 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8rmfb" podStartSLOduration=66.435132451 podStartE2EDuration="1m11.449283498s" totalImagesPullingTime="5.014151047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:49:23.448587128 +0000 UTC m=+71.820375703" watchObservedRunningTime="2026-07-20 17:49:23.449283498 +0000 UTC m=+71.821072073" Jul 20 17:49:23.469352 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:23.469306 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rshx6" podStartSLOduration=66.46428981 podStartE2EDuration="1m11.469292473s" totalImagesPullingTime="5.005002663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:49:23.468798694 +0000 UTC m=+71.840587270" watchObservedRunningTime="2026-07-20 17:49:23.469292473 +0000 UTC m=+71.841081046" Jul 20 17:49:29.792764 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:29.792712 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc"] Jul 20 17:49:29.794519 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:29.794504 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc" Jul 20 17:49:29.798879 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:29.798860 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 20 17:49:29.799829 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:29.799807 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 20 17:49:29.799939 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:29.799807 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-mtfw8\"" Jul 20 17:49:29.799939 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:29.799831 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 20 17:49:29.800518 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:29.800366 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 20 17:49:29.810173 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:29.810147 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc"] Jul 20 17:49:29.826669 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:29.826646 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7cc112b9-237b-4b0b-b009-a2416a7a8039-config\") pod \"service-ca-operator-5bcbc94c78-jslvc\" (UID: \"7cc112b9-237b-4b0b-b009-a2416a7a8039\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc" Jul 20 17:49:29.826773 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:29.826679 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8qkv\" (UniqueName: \"kubernetes.io/projected/7cc112b9-237b-4b0b-b009-a2416a7a8039-kube-api-access-n8qkv\") pod \"service-ca-operator-5bcbc94c78-jslvc\" (UID: \"7cc112b9-237b-4b0b-b009-a2416a7a8039\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc" Jul 20 17:49:29.826773 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:29.826704 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7cc112b9-237b-4b0b-b009-a2416a7a8039-serving-cert\") pod \"service-ca-operator-5bcbc94c78-jslvc\" (UID: \"7cc112b9-237b-4b0b-b009-a2416a7a8039\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc" Jul 20 17:49:29.927694 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:29.927661 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7cc112b9-237b-4b0b-b009-a2416a7a8039-config\") pod \"service-ca-operator-5bcbc94c78-jslvc\" (UID: \"7cc112b9-237b-4b0b-b009-a2416a7a8039\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc" Jul 20 17:49:29.927874 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:29.927698 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8qkv\" (UniqueName: \"kubernetes.io/projected/7cc112b9-237b-4b0b-b009-a2416a7a8039-kube-api-access-n8qkv\") pod \"service-ca-operator-5bcbc94c78-jslvc\" (UID: \"7cc112b9-237b-4b0b-b009-a2416a7a8039\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc" Jul 20 17:49:29.927874 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:29.927746 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7cc112b9-237b-4b0b-b009-a2416a7a8039-serving-cert\") pod \"service-ca-operator-5bcbc94c78-jslvc\" (UID: \"7cc112b9-237b-4b0b-b009-a2416a7a8039\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc" Jul 20 17:49:29.928260 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:29.928234 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7cc112b9-237b-4b0b-b009-a2416a7a8039-config\") pod \"service-ca-operator-5bcbc94c78-jslvc\" (UID: \"7cc112b9-237b-4b0b-b009-a2416a7a8039\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc" Jul 20 17:49:29.929870 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:29.929850 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7cc112b9-237b-4b0b-b009-a2416a7a8039-serving-cert\") pod \"service-ca-operator-5bcbc94c78-jslvc\" (UID: \"7cc112b9-237b-4b0b-b009-a2416a7a8039\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc" Jul 20 17:49:29.943512 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:29.943491 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8qkv\" (UniqueName: \"kubernetes.io/projected/7cc112b9-237b-4b0b-b009-a2416a7a8039-kube-api-access-n8qkv\") pod \"service-ca-operator-5bcbc94c78-jslvc\" (UID: \"7cc112b9-237b-4b0b-b009-a2416a7a8039\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc" Jul 20 17:49:30.062193 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.062100 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9"] Jul 20 17:49:30.063789 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.063768 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f"] Jul 20 17:49:30.063972 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.063883 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9" Jul 20 17:49:30.065526 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.065499 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f" Jul 20 17:49:30.072745 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.072703 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 20 17:49:30.072862 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.072745 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 20 17:49:30.072862 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.072855 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 20 17:49:30.072955 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.072885 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 20 17:49:30.073129 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.073111 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 20 17:49:30.073173 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.073117 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 20 17:49:30.073173 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.073162 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8cmxk\"" Jul 20 17:49:30.073245 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.073180 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rprzd\"" Jul 20 17:49:30.085538 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.085520 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 20 17:49:30.091890 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.091853 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9"] Jul 20 17:49:30.092465 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.092446 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f"] Jul 20 17:49:30.102804 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.102775 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc" Jul 20 17:49:30.129532 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.129501 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccm87\" (UniqueName: \"kubernetes.io/projected/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-kube-api-access-ccm87\") pod \"cluster-samples-operator-79b7c55f87-qks2f\" (UID: \"8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f" Jul 20 17:49:30.129532 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.129536 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f897e0b-72f4-40a1-917e-8e03196cedfe-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-xrcw9\" (UID: \"6f897e0b-72f4-40a1-917e-8e03196cedfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9" Jul 20 17:49:30.129701 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.129571 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f897e0b-72f4-40a1-917e-8e03196cedfe-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-xrcw9\" (UID: \"6f897e0b-72f4-40a1-917e-8e03196cedfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9" Jul 20 17:49:30.129701 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.129614 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h67d\" (UniqueName: \"kubernetes.io/projected/6f897e0b-72f4-40a1-917e-8e03196cedfe-kube-api-access-9h67d\") pod \"kube-storage-version-migrator-operator-7658d9b749-xrcw9\" (UID: \"6f897e0b-72f4-40a1-917e-8e03196cedfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9" Jul 20 17:49:30.129701 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.129648 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qks2f\" (UID: \"8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f" Jul 20 17:49:30.149116 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.149043 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-8v8sv"] Jul 20 17:49:30.151986 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.151961 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-97ccd8d99-4ckws"] Jul 20 17:49:30.152178 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.152079 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" Jul 20 17:49:30.154338 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.154320 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.157867 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.157852 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 20 17:49:30.158626 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.158313 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 20 17:49:30.159143 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.159120 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 20 17:49:30.159252 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.159124 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 20 17:49:30.159317 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.159267 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-shp6f\"" Jul 20 17:49:30.166756 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.165195 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 20 17:49:30.166756 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.165663 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 20 17:49:30.168335 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.167983 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-v4fzg\"" Jul 20 17:49:30.168714 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.168672 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 20 17:49:30.170959 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.170940 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 20 17:49:30.172749 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.172715 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 20 17:49:30.178814 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.178756 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-97ccd8d99-4ckws"] Jul 20 17:49:30.183392 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.183371 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-8v8sv"] Jul 20 17:49:30.230082 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.230051 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-ca-trust-extracted\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.230256 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.230090 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-certificates\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.230256 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.230115 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-bound-sa-token\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.230256 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.230168 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/084f381e-6cd8-4547-9de6-9808e8655048-serving-cert\") pod \"console-operator-565b7b569d-8v8sv\" (UID: \"084f381e-6cd8-4547-9de6-9808e8655048\") " pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" Jul 20 17:49:30.230256 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.230195 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8lgt\" (UniqueName: \"kubernetes.io/projected/084f381e-6cd8-4547-9de6-9808e8655048-kube-api-access-g8lgt\") pod \"console-operator-565b7b569d-8v8sv\" (UID: \"084f381e-6cd8-4547-9de6-9808e8655048\") " pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" Jul 20 17:49:30.230256 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.230248 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n47zd\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-kube-api-access-n47zd\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.230462 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.230299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccm87\" (UniqueName: \"kubernetes.io/projected/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-kube-api-access-ccm87\") pod \"cluster-samples-operator-79b7c55f87-qks2f\" (UID: \"8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f" Jul 20 17:49:30.230462 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.230318 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f897e0b-72f4-40a1-917e-8e03196cedfe-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-xrcw9\" (UID: \"6f897e0b-72f4-40a1-917e-8e03196cedfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9" Jul 20 17:49:30.230462 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.230342 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f897e0b-72f4-40a1-917e-8e03196cedfe-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-xrcw9\" (UID: \"6f897e0b-72f4-40a1-917e-8e03196cedfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9" Jul 20 17:49:30.230462 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.230358 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-trusted-ca\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.230462 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.230380 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h67d\" (UniqueName: \"kubernetes.io/projected/6f897e0b-72f4-40a1-917e-8e03196cedfe-kube-api-access-9h67d\") pod \"kube-storage-version-migrator-operator-7658d9b749-xrcw9\" (UID: \"6f897e0b-72f4-40a1-917e-8e03196cedfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9" Jul 20 17:49:30.230462 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.230398 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/084f381e-6cd8-4547-9de6-9808e8655048-trusted-ca\") pod \"console-operator-565b7b569d-8v8sv\" (UID: \"084f381e-6cd8-4547-9de6-9808e8655048\") " pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" Jul 20 17:49:30.230462 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.230412 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.230462 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.230438 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-image-registry-private-configuration\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.230868 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.230525 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/084f381e-6cd8-4547-9de6-9808e8655048-config\") pod \"console-operator-565b7b569d-8v8sv\" (UID: \"084f381e-6cd8-4547-9de6-9808e8655048\") " pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" Jul 20 17:49:30.230868 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.230563 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-installation-pull-secrets\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.230868 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.230600 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qks2f\" (UID: \"8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f" Jul 20 17:49:30.230868 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:30.230700 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 20 17:49:30.230868 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:30.230772 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-samples-operator-tls podName:8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d nodeName:}" failed. No retries permitted until 2026-07-20 17:49:30.730757654 +0000 UTC m=+79.102546209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-qks2f" (UID: "8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d") : secret "samples-operator-tls" not found Jul 20 17:49:30.231147 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.231126 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f897e0b-72f4-40a1-917e-8e03196cedfe-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-xrcw9\" (UID: \"6f897e0b-72f4-40a1-917e-8e03196cedfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9" Jul 20 17:49:30.232632 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.232612 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f897e0b-72f4-40a1-917e-8e03196cedfe-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-xrcw9\" (UID: \"6f897e0b-72f4-40a1-917e-8e03196cedfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9" Jul 20 17:49:30.242791 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.242702 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc"] Jul 20 17:49:30.246390 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:49:30.246370 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc112b9_237b_4b0b_b009_a2416a7a8039.slice/crio-f9fed4836e8575fe3bb81690811fcc06ba96c267680d74c7c80e23c6568053dd WatchSource:0}: Error finding container f9fed4836e8575fe3bb81690811fcc06ba96c267680d74c7c80e23c6568053dd: Status 404 returned error can't find the container with id f9fed4836e8575fe3bb81690811fcc06ba96c267680d74c7c80e23c6568053dd Jul 20 17:49:30.253377 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.253342 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h67d\" (UniqueName: \"kubernetes.io/projected/6f897e0b-72f4-40a1-917e-8e03196cedfe-kube-api-access-9h67d\") pod \"kube-storage-version-migrator-operator-7658d9b749-xrcw9\" (UID: \"6f897e0b-72f4-40a1-917e-8e03196cedfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9" Jul 20 17:49:30.253646 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.253627 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccm87\" (UniqueName: \"kubernetes.io/projected/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-kube-api-access-ccm87\") pod \"cluster-samples-operator-79b7c55f87-qks2f\" (UID: \"8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f" Jul 20 17:49:30.331434 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.331354 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-trusted-ca\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.331434 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.331394 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/084f381e-6cd8-4547-9de6-9808e8655048-trusted-ca\") pod \"console-operator-565b7b569d-8v8sv\" (UID: \"084f381e-6cd8-4547-9de6-9808e8655048\") " pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" Jul 20 17:49:30.331434 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.331418 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.331875 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.331846 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-image-registry-private-configuration\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.332023 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:30.331897 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 17:49:30.332023 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.331916 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/084f381e-6cd8-4547-9de6-9808e8655048-config\") pod \"console-operator-565b7b569d-8v8sv\" (UID: \"084f381e-6cd8-4547-9de6-9808e8655048\") " pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" Jul 20 17:49:30.332023 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:30.331925 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97ccd8d99-4ckws: secret "image-registry-tls" not found Jul 20 17:49:30.332023 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.331958 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-installation-pull-secrets\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.332023 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:30.332005 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls podName:a69e79ec-082f-47fc-ad1b-d64aa66c7e3f nodeName:}" failed. No retries permitted until 2026-07-20 17:49:30.831971432 +0000 UTC m=+79.203759990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls") pod "image-registry-97ccd8d99-4ckws" (UID: "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f") : secret "image-registry-tls" not found Jul 20 17:49:30.332284 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.332076 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-ca-trust-extracted\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.332284 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.332113 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-certificates\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.332284 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.332149 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-bound-sa-token\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.332284 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.332178 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/084f381e-6cd8-4547-9de6-9808e8655048-serving-cert\") pod \"console-operator-565b7b569d-8v8sv\" (UID: \"084f381e-6cd8-4547-9de6-9808e8655048\") " pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" Jul 20 17:49:30.332284 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.332213 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8lgt\" (UniqueName: \"kubernetes.io/projected/084f381e-6cd8-4547-9de6-9808e8655048-kube-api-access-g8lgt\") pod \"console-operator-565b7b569d-8v8sv\" (UID: \"084f381e-6cd8-4547-9de6-9808e8655048\") " pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" Jul 20 17:49:30.334291 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.332690 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-trusted-ca\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.334291 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.332706 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/084f381e-6cd8-4547-9de6-9808e8655048-trusted-ca\") pod \"console-operator-565b7b569d-8v8sv\" (UID: \"084f381e-6cd8-4547-9de6-9808e8655048\") " pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" Jul 20 17:49:30.334291 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.332958 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-certificates\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.334291 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.333262 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-ca-trust-extracted\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.334291 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.333363 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n47zd\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-kube-api-access-n47zd\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.334291 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.333447 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/084f381e-6cd8-4547-9de6-9808e8655048-config\") pod \"console-operator-565b7b569d-8v8sv\" (UID: \"084f381e-6cd8-4547-9de6-9808e8655048\") " pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" Jul 20 17:49:30.336129 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.335317 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/084f381e-6cd8-4547-9de6-9808e8655048-serving-cert\") pod \"console-operator-565b7b569d-8v8sv\" (UID: \"084f381e-6cd8-4547-9de6-9808e8655048\") " pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" Jul 20 17:49:30.336129 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.335514 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-installation-pull-secrets\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.337336 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.337318 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-image-registry-private-configuration\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.348359 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.348340 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-bound-sa-token\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.348987 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.348970 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n47zd\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-kube-api-access-n47zd\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.351264 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.351247 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8lgt\" (UniqueName: \"kubernetes.io/projected/084f381e-6cd8-4547-9de6-9808e8655048-kube-api-access-g8lgt\") pod \"console-operator-565b7b569d-8v8sv\" (UID: \"084f381e-6cd8-4547-9de6-9808e8655048\") " pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" Jul 20 17:49:30.375070 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.375050 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9" Jul 20 17:49:30.447860 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.447826 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc" event={"ID":"7cc112b9-237b-4b0b-b009-a2416a7a8039","Type":"ContainerStarted","Data":"f9fed4836e8575fe3bb81690811fcc06ba96c267680d74c7c80e23c6568053dd"} Jul 20 17:49:30.469221 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.469194 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" Jul 20 17:49:30.519388 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.519310 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9"] Jul 20 17:49:30.524258 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:49:30.524223 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f897e0b_72f4_40a1_917e_8e03196cedfe.slice/crio-ed4d4546fdfc637019bbe2fcec3411418468f3381530a947de502cb5b75f70d8 WatchSource:0}: Error finding container ed4d4546fdfc637019bbe2fcec3411418468f3381530a947de502cb5b75f70d8: Status 404 returned error can't find the container with id ed4d4546fdfc637019bbe2fcec3411418468f3381530a947de502cb5b75f70d8 Jul 20 17:49:30.627443 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.627368 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-8v8sv"] Jul 20 17:49:30.630934 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:49:30.630908 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084f381e_6cd8_4547_9de6_9808e8655048.slice/crio-b20258ef6c578221401a575d6efd746b2d01e8ac21614e0b7e688015200186e2 WatchSource:0}: Error finding container b20258ef6c578221401a575d6efd746b2d01e8ac21614e0b7e688015200186e2: Status 404 returned error can't find the container with id b20258ef6c578221401a575d6efd746b2d01e8ac21614e0b7e688015200186e2 Jul 20 17:49:30.737275 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.737246 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qks2f\" (UID: \"8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f" Jul 20 17:49:30.737417 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:30.737403 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 20 17:49:30.737472 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:30.737462 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-samples-operator-tls podName:8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d nodeName:}" failed. No retries permitted until 2026-07-20 17:49:31.737445928 +0000 UTC m=+80.109234500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-qks2f" (UID: "8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d") : secret "samples-operator-tls" not found Jul 20 17:49:30.837846 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:30.837821 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:30.838155 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:30.837962 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 17:49:30.838155 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:30.837977 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97ccd8d99-4ckws: secret "image-registry-tls" not found Jul 20 17:49:30.838155 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:30.838031 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls podName:a69e79ec-082f-47fc-ad1b-d64aa66c7e3f nodeName:}" failed. No retries permitted until 2026-07-20 17:49:31.83801437 +0000 UTC m=+80.209802941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls") pod "image-registry-97ccd8d99-4ckws" (UID: "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f") : secret "image-registry-tls" not found Jul 20 17:49:31.452770 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:31.452671 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" event={"ID":"084f381e-6cd8-4547-9de6-9808e8655048","Type":"ContainerStarted","Data":"b20258ef6c578221401a575d6efd746b2d01e8ac21614e0b7e688015200186e2"} Jul 20 17:49:31.454046 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:31.454004 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9" event={"ID":"6f897e0b-72f4-40a1-917e-8e03196cedfe","Type":"ContainerStarted","Data":"ed4d4546fdfc637019bbe2fcec3411418468f3381530a947de502cb5b75f70d8"} Jul 20 17:49:31.744253 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:31.744165 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qks2f\" (UID: \"8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f" Jul 20 17:49:31.744411 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:31.744347 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 20 17:49:31.744471 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:31.744428 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-samples-operator-tls podName:8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d nodeName:}" failed. No retries permitted until 2026-07-20 17:49:33.744407198 +0000 UTC m=+82.116195760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-qks2f" (UID: "8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d") : secret "samples-operator-tls" not found Jul 20 17:49:31.845637 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:31.845600 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:31.846113 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:31.845793 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 17:49:31.846113 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:31.845815 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97ccd8d99-4ckws: secret "image-registry-tls" not found Jul 20 17:49:31.846113 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:31.845882 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls podName:a69e79ec-082f-47fc-ad1b-d64aa66c7e3f nodeName:}" failed. No retries permitted until 2026-07-20 17:49:33.845862165 +0000 UTC m=+82.217650740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls") pod "image-registry-97ccd8d99-4ckws" (UID: "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f") : secret "image-registry-tls" not found Jul 20 17:49:33.759536 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:33.759420 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qks2f\" (UID: \"8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f" Jul 20 17:49:33.759536 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:33.759524 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 20 17:49:33.760036 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:33.759589 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-samples-operator-tls podName:8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d nodeName:}" failed. No retries permitted until 2026-07-20 17:49:37.759571443 +0000 UTC m=+86.131359996 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-qks2f" (UID: "8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d") : secret "samples-operator-tls" not found Jul 20 17:49:33.859770 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:33.859742 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:33.859885 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:33.859823 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 17:49:33.859885 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:33.859838 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97ccd8d99-4ckws: secret "image-registry-tls" not found Jul 20 17:49:33.859885 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:33.859884 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls podName:a69e79ec-082f-47fc-ad1b-d64aa66c7e3f nodeName:}" failed. No retries permitted until 2026-07-20 17:49:37.859870496 +0000 UTC m=+86.231659052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls") pod "image-registry-97ccd8d99-4ckws" (UID: "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f") : secret "image-registry-tls" not found Jul 20 17:49:34.445053 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.445021 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gkzdq"] Jul 20 17:49:34.448303 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.448288 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:34.457279 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.457258 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 20 17:49:34.458008 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.457989 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 20 17:49:34.458230 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.458217 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 20 17:49:34.459529 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.459510 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hq7ht\"" Jul 20 17:49:34.460902 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.460885 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 20 17:49:34.463800 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.463777 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8490308-78b2-45e7-b449-2126c3cd30a9-data-volume\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:34.463884 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.463808 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc" event={"ID":"7cc112b9-237b-4b0b-b009-a2416a7a8039","Type":"ContainerStarted","Data":"7d445d4cb25d059faf32b5fe303d5b7416908adca51e76a7059d91f658cf572f"} Jul 20 17:49:34.463884 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.463859 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:34.464016 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.463886 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4qjx\" (UniqueName: \"kubernetes.io/projected/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-api-access-q4qjx\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:34.464016 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.463918 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:34.464016 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.463985 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e8490308-78b2-45e7-b449-2126c3cd30a9-crio-socket\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:34.465375 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.465352 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9" event={"ID":"6f897e0b-72f4-40a1-917e-8e03196cedfe","Type":"ContainerStarted","Data":"3840df42a4a0d353262256cca0bbe6b886159100537aa116d98326df6c452303"} Jul 20 17:49:34.466956 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.466939 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/0.log" Jul 20 17:49:34.467030 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.466977 2583 generic.go:356] "Generic (PLEG): container finished" podID="084f381e-6cd8-4547-9de6-9808e8655048" containerID="de7e99bc61571aba053bb914f68bcdb816327f4201a577532cbb2fe4a3e15c5e" exitCode=255 Jul 20 17:49:34.467030 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.467008 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" event={"ID":"084f381e-6cd8-4547-9de6-9808e8655048","Type":"ContainerDied","Data":"de7e99bc61571aba053bb914f68bcdb816327f4201a577532cbb2fe4a3e15c5e"} Jul 20 17:49:34.467179 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.467167 2583 scope.go:117] "RemoveContainer" containerID="de7e99bc61571aba053bb914f68bcdb816327f4201a577532cbb2fe4a3e15c5e" Jul 20 17:49:34.471410 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.471323 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gkzdq"] Jul 20 17:49:34.562306 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.562245 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc" podStartSLOduration=2.000427525 podStartE2EDuration="5.562222914s" totalImagesPullingTime="3.561795389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:49:34.562115154 +0000 UTC m=+82.933903729" watchObservedRunningTime="2026-07-20 17:49:34.562222914 +0000 UTC m=+82.934011489" Jul 20 17:49:34.564364 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.564323 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8490308-78b2-45e7-b449-2126c3cd30a9-data-volume\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:34.564993 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.564968 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:34.565129 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.565005 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4qjx\" (UniqueName: \"kubernetes.io/projected/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-api-access-q4qjx\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:34.565129 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.565059 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:34.565244 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:34.565141 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 17:49:34.565244 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.565170 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e8490308-78b2-45e7-b449-2126c3cd30a9-crio-socket\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:34.565244 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:34.565210 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-rbac-proxy-cm podName:e8490308-78b2-45e7-b449-2126c3cd30a9 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:35.065193688 +0000 UTC m=+83.436982256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gkzdq" (UID: "e8490308-78b2-45e7-b449-2126c3cd30a9") : configmap "kube-rbac-proxy" not found Jul 20 17:49:34.565403 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.565273 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e8490308-78b2-45e7-b449-2126c3cd30a9-crio-socket\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:34.565453 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:34.565432 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 17:49:34.565498 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:34.565478 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls podName:e8490308-78b2-45e7-b449-2126c3cd30a9 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:35.065464814 +0000 UTC m=+83.437253365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gkzdq" (UID: "e8490308-78b2-45e7-b449-2126c3cd30a9") : secret "insights-runtime-extractor-tls" not found Jul 20 17:49:34.565690 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.565667 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8490308-78b2-45e7-b449-2126c3cd30a9-data-volume\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:34.591334 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.591308 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4qjx\" (UniqueName: \"kubernetes.io/projected/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-api-access-q4qjx\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:34.640596 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:34.640545 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9" podStartSLOduration=2.353841194 podStartE2EDuration="5.640527042s" totalImagesPullingTime="3.286685848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:49:34.639297588 +0000 UTC m=+83.011086163" watchObservedRunningTime="2026-07-20 17:49:34.640527042 +0000 UTC m=+83.012315673" Jul 20 17:49:35.069248 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.069208 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:35.069248 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.069248 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:35.069667 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:35.069333 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 17:49:35.069667 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:35.069356 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 17:49:35.069667 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:35.069405 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-rbac-proxy-cm podName:e8490308-78b2-45e7-b449-2126c3cd30a9 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:36.069388908 +0000 UTC m=+84.441177465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gkzdq" (UID: "e8490308-78b2-45e7-b449-2126c3cd30a9") : configmap "kube-rbac-proxy" not found Jul 20 17:49:35.069667 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:35.069419 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls podName:e8490308-78b2-45e7-b449-2126c3cd30a9 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:36.069413547 +0000 UTC m=+84.441202099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gkzdq" (UID: "e8490308-78b2-45e7-b449-2126c3cd30a9") : secret "insights-runtime-extractor-tls" not found Jul 20 17:49:35.467656 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.467628 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bcrpq"] Jul 20 17:49:35.470643 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.470621 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bcrpq" Jul 20 17:49:35.472957 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.472935 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 17:49:35.473357 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.473344 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/0.log" Jul 20 17:49:35.473442 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.473373 2583 generic.go:356] "Generic (PLEG): container finished" podID="084f381e-6cd8-4547-9de6-9808e8655048" containerID="3f2f973513a2ff55650f2ae0a4cca59207a9b616aaf20d7028c3b0a735c7257b" exitCode=255 Jul 20 17:49:35.473494 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.473446 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" event={"ID":"084f381e-6cd8-4547-9de6-9808e8655048","Type":"ContainerDied","Data":"3f2f973513a2ff55650f2ae0a4cca59207a9b616aaf20d7028c3b0a735c7257b"} Jul 20 17:49:35.473546 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.473499 2583 scope.go:117] "RemoveContainer" containerID="de7e99bc61571aba053bb914f68bcdb816327f4201a577532cbb2fe4a3e15c5e" Jul 20 17:49:35.473799 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.473766 2583 scope.go:117] "RemoveContainer" containerID="3f2f973513a2ff55650f2ae0a4cca59207a9b616aaf20d7028c3b0a735c7257b" Jul 20 17:49:35.473961 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:35.473942 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-8v8sv_openshift-console-operator(084f381e-6cd8-4547-9de6-9808e8655048)\"" pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" podUID="084f381e-6cd8-4547-9de6-9808e8655048" Jul 20 17:49:35.475295 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.475279 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-54rsl\"" Jul 20 17:49:35.480950 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.480932 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 20 17:49:35.483600 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.483584 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 20 17:49:35.488564 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.488544 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bcrpq"] Jul 20 17:49:35.572614 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.572573 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zthlt\" (UniqueName: \"kubernetes.io/projected/0194534a-df67-4bad-a569-0e453aed46a2-kube-api-access-zthlt\") pod \"migrator-7cc9c7ffd7-bcrpq\" (UID: \"0194534a-df67-4bad-a569-0e453aed46a2\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bcrpq" Jul 20 17:49:35.673549 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.673512 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zthlt\" (UniqueName: \"kubernetes.io/projected/0194534a-df67-4bad-a569-0e453aed46a2-kube-api-access-zthlt\") pod \"migrator-7cc9c7ffd7-bcrpq\" (UID: \"0194534a-df67-4bad-a569-0e453aed46a2\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bcrpq" Jul 20 17:49:35.683601 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.683578 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zthlt\" (UniqueName: \"kubernetes.io/projected/0194534a-df67-4bad-a569-0e453aed46a2-kube-api-access-zthlt\") pod \"migrator-7cc9c7ffd7-bcrpq\" (UID: \"0194534a-df67-4bad-a569-0e453aed46a2\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bcrpq" Jul 20 17:49:35.780787 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.780681 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bcrpq" Jul 20 17:49:35.909609 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:35.909567 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bcrpq"] Jul 20 17:49:35.912953 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:49:35.912926 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0194534a_df67_4bad_a569_0e453aed46a2.slice/crio-b6db695d4ffd9beef5dbdd6f2f1260b920d8611dff223f30ffd40d0f88d843af WatchSource:0}: Error finding container b6db695d4ffd9beef5dbdd6f2f1260b920d8611dff223f30ffd40d0f88d843af: Status 404 returned error can't find the container with id b6db695d4ffd9beef5dbdd6f2f1260b920d8611dff223f30ffd40d0f88d843af Jul 20 17:49:36.076260 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:36.076180 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:36.076260 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:36.076220 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:36.076634 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:36.076324 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 17:49:36.076634 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:36.076381 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls podName:e8490308-78b2-45e7-b449-2126c3cd30a9 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:38.076365962 +0000 UTC m=+86.448154514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gkzdq" (UID: "e8490308-78b2-45e7-b449-2126c3cd30a9") : secret "insights-runtime-extractor-tls" not found Jul 20 17:49:36.076634 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:36.076333 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 17:49:36.076634 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:36.076444 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-rbac-proxy-cm podName:e8490308-78b2-45e7-b449-2126c3cd30a9 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:38.076432897 +0000 UTC m=+86.448221452 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gkzdq" (UID: "e8490308-78b2-45e7-b449-2126c3cd30a9") : configmap "kube-rbac-proxy" not found Jul 20 17:49:36.478791 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:36.478754 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bcrpq" event={"ID":"0194534a-df67-4bad-a569-0e453aed46a2","Type":"ContainerStarted","Data":"b6db695d4ffd9beef5dbdd6f2f1260b920d8611dff223f30ffd40d0f88d843af"} Jul 20 17:49:36.480647 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:36.480624 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 17:49:36.481021 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:36.481001 2583 scope.go:117] "RemoveContainer" containerID="3f2f973513a2ff55650f2ae0a4cca59207a9b616aaf20d7028c3b0a735c7257b" Jul 20 17:49:36.481191 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:36.481171 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-8v8sv_openshift-console-operator(084f381e-6cd8-4547-9de6-9808e8655048)\"" pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" podUID="084f381e-6cd8-4547-9de6-9808e8655048" Jul 20 17:49:36.493261 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:36.493241 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qgk6s_167b1498-09ad-4ab3-84e9-8787235afb26/dns-node-resolver/0.log" Jul 20 17:49:37.486347 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:37.486256 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bcrpq" event={"ID":"0194534a-df67-4bad-a569-0e453aed46a2","Type":"ContainerStarted","Data":"2ba19120de66ec31ef6c2dba44272efb7336c96c1b0068b8672c55e0a2c37fb9"} Jul 20 17:49:37.486347 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:37.486316 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bcrpq" event={"ID":"0194534a-df67-4bad-a569-0e453aed46a2","Type":"ContainerStarted","Data":"72a7d67e2deb5e94b13649c59e6bd8169088516de25280d4f292bdf20c2c7fea"} Jul 20 17:49:37.502914 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:37.502891 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7jrr6_0ff18797-615b-4cc4-82f2-3621080f562e/node-ca/0.log" Jul 20 17:49:37.788836 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:37.788743 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qks2f\" (UID: \"8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f" Jul 20 17:49:37.788986 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:37.788868 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 20 17:49:37.788986 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:37.788921 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-samples-operator-tls podName:8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d nodeName:}" failed. No retries permitted until 2026-07-20 17:49:45.788906993 +0000 UTC m=+94.160695544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-qks2f" (UID: "8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d") : secret "samples-operator-tls" not found Jul 20 17:49:37.889720 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:37.889683 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:37.889876 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:37.889844 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 17:49:37.889876 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:37.889862 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97ccd8d99-4ckws: secret "image-registry-tls" not found Jul 20 17:49:37.889973 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:37.889911 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls podName:a69e79ec-082f-47fc-ad1b-d64aa66c7e3f nodeName:}" failed. No retries permitted until 2026-07-20 17:49:45.889898176 +0000 UTC m=+94.261686731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls") pod "image-registry-97ccd8d99-4ckws" (UID: "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f") : secret "image-registry-tls" not found Jul 20 17:49:38.091663 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:38.091574 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:38.091663 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:38.091617 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:38.091873 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:38.091693 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 17:49:38.091873 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:38.091720 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 17:49:38.091873 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:38.091769 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-rbac-proxy-cm podName:e8490308-78b2-45e7-b449-2126c3cd30a9 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:42.091755808 +0000 UTC m=+90.463544360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gkzdq" (UID: "e8490308-78b2-45e7-b449-2126c3cd30a9") : configmap "kube-rbac-proxy" not found Jul 20 17:49:38.091873 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:38.091795 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls podName:e8490308-78b2-45e7-b449-2126c3cd30a9 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:42.091778661 +0000 UTC m=+90.463567218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gkzdq" (UID: "e8490308-78b2-45e7-b449-2126c3cd30a9") : secret "insights-runtime-extractor-tls" not found Jul 20 17:49:38.291681 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:38.291647 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9rqgb_5c931ca0-fe6b-4e41-8bf8-53617767759f/serve-healthcheck-canary/0.log" Jul 20 17:49:40.469297 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:40.469259 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" Jul 20 17:49:40.469682 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:40.469369 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" Jul 20 17:49:40.469682 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:40.469673 2583 scope.go:117] "RemoveContainer" containerID="3f2f973513a2ff55650f2ae0a4cca59207a9b616aaf20d7028c3b0a735c7257b" Jul 20 17:49:40.469861 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:40.469842 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-8v8sv_openshift-console-operator(084f381e-6cd8-4547-9de6-9808e8655048)\"" pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" podUID="084f381e-6cd8-4547-9de6-9808e8655048" Jul 20 17:49:42.124901 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:42.124860 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:42.124901 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:42.124906 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:42.125326 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:42.124995 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 17:49:42.125326 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:42.125015 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 17:49:42.125326 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:42.125056 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-rbac-proxy-cm podName:e8490308-78b2-45e7-b449-2126c3cd30a9 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:50.125042456 +0000 UTC m=+98.496831008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gkzdq" (UID: "e8490308-78b2-45e7-b449-2126c3cd30a9") : configmap "kube-rbac-proxy" not found Jul 20 17:49:42.125326 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:42.125078 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls podName:e8490308-78b2-45e7-b449-2126c3cd30a9 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:50.125064377 +0000 UTC m=+98.496852930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gkzdq" (UID: "e8490308-78b2-45e7-b449-2126c3cd30a9") : secret "insights-runtime-extractor-tls" not found Jul 20 17:49:45.854045 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:45.854007 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qks2f\" (UID: \"8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f" Jul 20 17:49:45.856308 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:45.856288 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qks2f\" (UID: \"8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f" Jul 20 17:49:45.954708 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:45.954675 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:45.957027 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:45.956997 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls\") pod \"image-registry-97ccd8d99-4ckws\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:45.979889 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:45.979867 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f" Jul 20 17:49:46.073996 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:46.073739 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:46.112769 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:46.112702 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bcrpq" podStartSLOduration=9.829373747 podStartE2EDuration="11.112681773s" totalImagesPullingTime="1.283308026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:49:37.527132771 +0000 UTC m=+85.898921345" watchObservedRunningTime="2026-07-20 17:49:46.112681773 +0000 UTC m=+94.484470349" Jul 20 17:49:46.112886 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:46.112862 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f"] Jul 20 17:49:46.211549 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:46.211525 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-97ccd8d99-4ckws"] Jul 20 17:49:46.213576 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:49:46.213546 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69e79ec_082f_47fc_ad1b_d64aa66c7e3f.slice/crio-cd5dc95135ef965802573463f513421dced55414e95368e2a761b67672d7480e WatchSource:0}: Error finding container cd5dc95135ef965802573463f513421dced55414e95368e2a761b67672d7480e: Status 404 returned error can't find the container with id cd5dc95135ef965802573463f513421dced55414e95368e2a761b67672d7480e Jul 20 17:49:46.514309 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:46.514271 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f" event={"ID":"8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d","Type":"ContainerStarted","Data":"5cc997c3817b134c0c44a68bb7601d189237be432fdadb57bdf04776962b112b"} Jul 20 17:49:46.515767 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:46.515713 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" event={"ID":"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f","Type":"ContainerStarted","Data":"7140176b33b0063162304e3e5cb53373fe4aebb6426e6a0f9e2664f08b7bb3ed"} Jul 20 17:49:46.515918 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:46.515784 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" event={"ID":"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f","Type":"ContainerStarted","Data":"cd5dc95135ef965802573463f513421dced55414e95368e2a761b67672d7480e"} Jul 20 17:49:46.515918 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:46.515827 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:49:48.523873 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:48.523838 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f" event={"ID":"8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d","Type":"ContainerStarted","Data":"abe73ed7e28583e70e12453196c9431af5c8f26889a2e1db9cbf965c56c0fbd8"} Jul 20 17:49:48.524270 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:48.523896 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f" event={"ID":"8f5ffa76-ccae-4f6d-b39b-8e6ed470bb3d","Type":"ContainerStarted","Data":"40726f06d478cdd2de5a44a98046ae4f9a9cc4476f98c4f33cb169a9aaad3324"} Jul 20 17:49:48.552576 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:48.552526 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" podStartSLOduration=18.552508994 podStartE2EDuration="18.552508994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:49:46.553550872 +0000 UTC m=+94.925339471" watchObservedRunningTime="2026-07-20 17:49:48.552508994 +0000 UTC m=+96.924297569" Jul 20 17:49:48.553152 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:48.553123 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qks2f" podStartSLOduration=17.683227056 podStartE2EDuration="19.553112187s" totalImagesPullingTime="1.869885131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:49:48.551141787 +0000 UTC m=+96.922930363" watchObservedRunningTime="2026-07-20 17:49:48.553112187 +0000 UTC m=+96.924900761" Jul 20 17:49:50.186689 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:50.186653 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:50.187171 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:50.186888 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:50.187171 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:50.187001 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 17:49:50.187171 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:49:50.187069 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls podName:e8490308-78b2-45e7-b449-2126c3cd30a9 nodeName:}" failed. No retries permitted until 2026-07-20 17:50:06.187051483 +0000 UTC m=+114.558840038 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gkzdq" (UID: "e8490308-78b2-45e7-b449-2126c3cd30a9") : secret "insights-runtime-extractor-tls" not found Jul 20 17:49:50.187349 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:50.187195 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8490308-78b2-45e7-b449-2126c3cd30a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:49:50.389158 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:50.389114 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:49:50.391479 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:50.391453 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9477e1ed-8e61-4f89-8cc3-db20326fce10-metrics-tls\") pod \"dns-default-qvw7j\" (UID: \"9477e1ed-8e61-4f89-8cc3-db20326fce10\") " pod="openshift-dns/dns-default-qvw7j" Jul 20 17:49:50.481439 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:50.481353 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nkvwc\"" Jul 20 17:49:50.488574 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:50.488552 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qvw7j" Jul 20 17:49:50.631415 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:50.631386 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qvw7j"] Jul 20 17:49:50.637364 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:49:50.637332 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9477e1ed_8e61_4f89_8cc3_db20326fce10.slice/crio-7f252de35b291d2cede2f6d6a3f04cb9469391ecfb6d4aa250df832b13e124c3 WatchSource:0}: Error finding container 7f252de35b291d2cede2f6d6a3f04cb9469391ecfb6d4aa250df832b13e124c3: Status 404 returned error can't find the container with id 7f252de35b291d2cede2f6d6a3f04cb9469391ecfb6d4aa250df832b13e124c3 Jul 20 17:49:51.535607 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:51.535564 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qvw7j" event={"ID":"9477e1ed-8e61-4f89-8cc3-db20326fce10","Type":"ContainerStarted","Data":"7f252de35b291d2cede2f6d6a3f04cb9469391ecfb6d4aa250df832b13e124c3"} Jul 20 17:49:53.165508 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:53.165426 2583 scope.go:117] "RemoveContainer" containerID="3f2f973513a2ff55650f2ae0a4cca59207a9b616aaf20d7028c3b0a735c7257b" Jul 20 17:49:53.544229 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:53.544193 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qvw7j" event={"ID":"9477e1ed-8e61-4f89-8cc3-db20326fce10","Type":"ContainerStarted","Data":"04f79dbdb6d64d7f59ad218058ebb97973115c6e05c776fdad6ea2d02d260203"} Jul 20 17:49:53.544431 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:53.544268 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qvw7j" event={"ID":"9477e1ed-8e61-4f89-8cc3-db20326fce10","Type":"ContainerStarted","Data":"c324a7ba62824d8cf21ca5f2935fadd469585dcfe1a891c18ea74d0f16a59107"} Jul 20 17:49:53.544431 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:53.544325 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qvw7j" Jul 20 17:49:53.546138 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:53.546120 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 17:49:53.546237 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:53.546175 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" event={"ID":"084f381e-6cd8-4547-9de6-9808e8655048","Type":"ContainerStarted","Data":"6b429f438029af1857309bdeede658100e90e4dcd2292b5884c41f7f8486b0b2"} Jul 20 17:49:53.546430 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:53.546405 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" Jul 20 17:49:53.583583 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:53.583528 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qvw7j" podStartSLOduration=65.332711727 podStartE2EDuration="1m7.583508994s" totalImagesPullingTime="2.250797267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:49:53.564566713 +0000 UTC m=+101.936355288" watchObservedRunningTime="2026-07-20 17:49:53.583508994 +0000 UTC m=+101.955297569" Jul 20 17:49:53.584617 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:53.584574 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" podStartSLOduration=20.401510382 podStartE2EDuration="23.584558147s" totalImagesPullingTime="3.183047765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:49:53.583144952 +0000 UTC m=+101.954933528" watchObservedRunningTime="2026-07-20 17:49:53.584558147 +0000 UTC m=+101.956346724" Jul 20 17:49:53.722448 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:53.722416 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-8v8sv" Jul 20 17:49:54.435126 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:49:54.435094 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8rmfb" Jul 20 17:50:03.252613 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.252574 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-pz8zz"] Jul 20 17:50:03.254937 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.254916 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pz8zz" Jul 20 17:50:03.259152 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.259131 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 20 17:50:03.260005 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.259989 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 20 17:50:03.260097 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.259991 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6zhcl\"" Jul 20 17:50:03.271361 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.271334 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-pz8zz"] Jul 20 17:50:03.279022 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.279000 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-97ccd8d99-4ckws"] Jul 20 17:50:03.283165 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.283140 2583 patch_prober.go:28] interesting pod/image-registry-97ccd8d99-4ckws container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 20 17:50:03.283284 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.283189 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" podUID="a69e79ec-082f-47fc-ad1b-d64aa66c7e3f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 17:50:03.387278 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.387242 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f302722f-c610-42f7-972e-cb6885002372-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pz8zz\" (UID: \"f302722f-c610-42f7-972e-cb6885002372\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pz8zz" Jul 20 17:50:03.387278 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.387288 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f302722f-c610-42f7-972e-cb6885002372-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-pz8zz\" (UID: \"f302722f-c610-42f7-972e-cb6885002372\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pz8zz" Jul 20 17:50:03.488606 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.488570 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f302722f-c610-42f7-972e-cb6885002372-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pz8zz\" (UID: \"f302722f-c610-42f7-972e-cb6885002372\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pz8zz" Jul 20 17:50:03.488606 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.488614 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f302722f-c610-42f7-972e-cb6885002372-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-pz8zz\" (UID: \"f302722f-c610-42f7-972e-cb6885002372\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pz8zz" Jul 20 17:50:03.489370 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.489352 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f302722f-c610-42f7-972e-cb6885002372-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-pz8zz\" (UID: \"f302722f-c610-42f7-972e-cb6885002372\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pz8zz" Jul 20 17:50:03.490988 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.490968 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f302722f-c610-42f7-972e-cb6885002372-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pz8zz\" (UID: \"f302722f-c610-42f7-972e-cb6885002372\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pz8zz" Jul 20 17:50:03.552212 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.552140 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qvw7j" Jul 20 17:50:03.559940 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.559917 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cdfcc7df9-f7lqq"] Jul 20 17:50:03.562166 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.562148 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.564180 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.564161 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pz8zz" Jul 20 17:50:03.587641 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.587614 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cdfcc7df9-f7lqq"] Jul 20 17:50:03.690987 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.690942 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-bound-sa-token\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.691157 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.691051 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-registry-certificates\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.691222 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.691157 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-ca-trust-extracted\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.691222 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.691194 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-trusted-ca\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.691304 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.691263 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-installation-pull-secrets\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.691304 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.691294 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df4qc\" (UniqueName: \"kubernetes.io/projected/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-kube-api-access-df4qc\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.691388 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.691327 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-image-registry-private-configuration\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.691388 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.691376 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-registry-tls\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.720018 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.719999 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-pz8zz"] Jul 20 17:50:03.722126 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:50:03.722100 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf302722f_c610_42f7_972e_cb6885002372.slice/crio-1135a9fadece17de5a902e2be5ec470ab34cb8c9f6a081837a551c2c56e35c04 WatchSource:0}: Error finding container 1135a9fadece17de5a902e2be5ec470ab34cb8c9f6a081837a551c2c56e35c04: Status 404 returned error can't find the container with id 1135a9fadece17de5a902e2be5ec470ab34cb8c9f6a081837a551c2c56e35c04 Jul 20 17:50:03.791871 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.791841 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-registry-tls\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.792030 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.791878 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-bound-sa-token\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.792030 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.791910 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-registry-certificates\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.792030 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.792007 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-ca-trust-extracted\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.792193 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.792043 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-trusted-ca\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.792193 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.792068 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-installation-pull-secrets\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.792193 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.792091 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df4qc\" (UniqueName: \"kubernetes.io/projected/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-kube-api-access-df4qc\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.792374 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.792336 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-image-registry-private-configuration\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.792518 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.792388 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-ca-trust-extracted\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.792799 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.792778 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-registry-certificates\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.792943 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.792926 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-trusted-ca\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.794425 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.794396 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-installation-pull-secrets\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.794716 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.794699 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-registry-tls\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.794870 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.794850 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-image-registry-private-configuration\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.819151 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.819101 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df4qc\" (UniqueName: \"kubernetes.io/projected/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-kube-api-access-df4qc\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.822438 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.822418 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08c5249a-a8fe-47c5-816d-a8ed99cd2f44-bound-sa-token\") pod \"image-registry-7cdfcc7df9-f7lqq\" (UID: \"08c5249a-a8fe-47c5-816d-a8ed99cd2f44\") " pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:03.870713 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:03.870689 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:04.022075 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:04.022034 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cdfcc7df9-f7lqq"] Jul 20 17:50:04.025164 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:50:04.025130 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c5249a_a8fe_47c5_816d_a8ed99cd2f44.slice/crio-8f8566e99dd65e5a7742e41b365454b12f4e07147f98d90d581c9f194917c5a9 WatchSource:0}: Error finding container 8f8566e99dd65e5a7742e41b365454b12f4e07147f98d90d581c9f194917c5a9: Status 404 returned error can't find the container with id 8f8566e99dd65e5a7742e41b365454b12f4e07147f98d90d581c9f194917c5a9 Jul 20 17:50:04.585097 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:04.585046 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" event={"ID":"08c5249a-a8fe-47c5-816d-a8ed99cd2f44","Type":"ContainerStarted","Data":"0a5804d1997b2e21ef5ca8544567c51f3c604e0caecb00ce6a6543967f5b9926"} Jul 20 17:50:04.585607 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:04.585103 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" event={"ID":"08c5249a-a8fe-47c5-816d-a8ed99cd2f44","Type":"ContainerStarted","Data":"8f8566e99dd65e5a7742e41b365454b12f4e07147f98d90d581c9f194917c5a9"} Jul 20 17:50:04.585607 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:04.585250 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:04.586262 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:04.586237 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pz8zz" event={"ID":"f302722f-c610-42f7-972e-cb6885002372","Type":"ContainerStarted","Data":"1135a9fadece17de5a902e2be5ec470ab34cb8c9f6a081837a551c2c56e35c04"} Jul 20 17:50:04.633639 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:04.633587 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" podStartSLOduration=1.63356486 podStartE2EDuration="1.63356486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:50:04.631811677 +0000 UTC m=+113.003600468" watchObservedRunningTime="2026-07-20 17:50:04.63356486 +0000 UTC m=+113.005353412" Jul 20 17:50:05.592746 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:05.592701 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pz8zz" event={"ID":"f302722f-c610-42f7-972e-cb6885002372","Type":"ContainerStarted","Data":"229f46bde5525e19868b08e62129735f98076a2ee9b34f98e53b491c0a3be552"} Jul 20 17:50:05.618426 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:05.618377 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pz8zz" podStartSLOduration=0.950273563 podStartE2EDuration="2.618358714s" totalImagesPullingTime="1.668085151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:50:05.61657565 +0000 UTC m=+113.988364224" watchObservedRunningTime="2026-07-20 17:50:05.618358714 +0000 UTC m=+113.990147288" Jul 20 17:50:06.212776 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:06.212720 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:50:06.215253 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:06.215224 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8490308-78b2-45e7-b449-2126c3cd30a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkzdq\" (UID: \"e8490308-78b2-45e7-b449-2126c3cd30a9\") " pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:50:06.257239 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:06.257205 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gkzdq" Jul 20 17:50:06.402821 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:06.402791 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gkzdq"] Jul 20 17:50:06.406055 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:50:06.406013 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8490308_78b2_45e7_b449_2126c3cd30a9.slice/crio-274b9c29dd47d46c593d6ad78243bdf341afd1db63b16fda231965b34e638528 WatchSource:0}: Error finding container 274b9c29dd47d46c593d6ad78243bdf341afd1db63b16fda231965b34e638528: Status 404 returned error can't find the container with id 274b9c29dd47d46c593d6ad78243bdf341afd1db63b16fda231965b34e638528 Jul 20 17:50:06.596692 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:06.596600 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gkzdq" event={"ID":"e8490308-78b2-45e7-b449-2126c3cd30a9","Type":"ContainerStarted","Data":"274b9c29dd47d46c593d6ad78243bdf341afd1db63b16fda231965b34e638528"} Jul 20 17:50:10.500783 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.500721 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg"] Jul 20 17:50:10.522251 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.522220 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg" Jul 20 17:50:10.528619 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.528428 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 20 17:50:10.528619 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.528488 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 20 17:50:10.528619 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.528504 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 20 17:50:10.528619 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.528428 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vnjxj\"" Jul 20 17:50:10.528619 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.528571 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 20 17:50:10.528619 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.528493 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 20 17:50:10.533238 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.533172 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg"] Jul 20 17:50:10.645765 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.645708 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5rfn\" (UniqueName: \"kubernetes.io/projected/8a474b76-8147-4f5a-8bf6-933ca8943292-kube-api-access-c5rfn\") pod \"prometheus-operator-5945b5cfd-7nzqg\" (UID: \"8a474b76-8147-4f5a-8bf6-933ca8943292\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg" Jul 20 17:50:10.645961 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.645809 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a474b76-8147-4f5a-8bf6-933ca8943292-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-7nzqg\" (UID: \"8a474b76-8147-4f5a-8bf6-933ca8943292\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg" Jul 20 17:50:10.645961 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.645831 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a474b76-8147-4f5a-8bf6-933ca8943292-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-7nzqg\" (UID: \"8a474b76-8147-4f5a-8bf6-933ca8943292\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg" Jul 20 17:50:10.645961 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.645860 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a474b76-8147-4f5a-8bf6-933ca8943292-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-7nzqg\" (UID: \"8a474b76-8147-4f5a-8bf6-933ca8943292\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg" Jul 20 17:50:10.746507 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.746472 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5rfn\" (UniqueName: \"kubernetes.io/projected/8a474b76-8147-4f5a-8bf6-933ca8943292-kube-api-access-c5rfn\") pod \"prometheus-operator-5945b5cfd-7nzqg\" (UID: \"8a474b76-8147-4f5a-8bf6-933ca8943292\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg" Jul 20 17:50:10.746697 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.746533 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a474b76-8147-4f5a-8bf6-933ca8943292-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-7nzqg\" (UID: \"8a474b76-8147-4f5a-8bf6-933ca8943292\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg" Jul 20 17:50:10.746697 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.746553 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a474b76-8147-4f5a-8bf6-933ca8943292-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-7nzqg\" (UID: \"8a474b76-8147-4f5a-8bf6-933ca8943292\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg" Jul 20 17:50:10.746697 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.746590 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a474b76-8147-4f5a-8bf6-933ca8943292-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-7nzqg\" (UID: \"8a474b76-8147-4f5a-8bf6-933ca8943292\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg" Jul 20 17:50:10.747343 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.747322 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a474b76-8147-4f5a-8bf6-933ca8943292-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-7nzqg\" (UID: \"8a474b76-8147-4f5a-8bf6-933ca8943292\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg" Jul 20 17:50:10.749003 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.748976 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a474b76-8147-4f5a-8bf6-933ca8943292-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-7nzqg\" (UID: \"8a474b76-8147-4f5a-8bf6-933ca8943292\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg" Jul 20 17:50:10.749100 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.749060 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a474b76-8147-4f5a-8bf6-933ca8943292-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-7nzqg\" (UID: \"8a474b76-8147-4f5a-8bf6-933ca8943292\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg" Jul 20 17:50:10.767482 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.767412 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5rfn\" (UniqueName: \"kubernetes.io/projected/8a474b76-8147-4f5a-8bf6-933ca8943292-kube-api-access-c5rfn\") pod \"prometheus-operator-5945b5cfd-7nzqg\" (UID: \"8a474b76-8147-4f5a-8bf6-933ca8943292\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg" Jul 20 17:50:10.834879 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:10.834846 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg" Jul 20 17:50:11.603631 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:11.603480 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg"] Jul 20 17:50:11.606002 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:50:11.605974 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a474b76_8147_4f5a_8bf6_933ca8943292.slice/crio-239e853a196c0e1ad34b5ea8d0e062dc9af22f29cebebd66eebe054bf1b2ef3e WatchSource:0}: Error finding container 239e853a196c0e1ad34b5ea8d0e062dc9af22f29cebebd66eebe054bf1b2ef3e: Status 404 returned error can't find the container with id 239e853a196c0e1ad34b5ea8d0e062dc9af22f29cebebd66eebe054bf1b2ef3e Jul 20 17:50:11.615050 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:11.615022 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg" event={"ID":"8a474b76-8147-4f5a-8bf6-933ca8943292","Type":"ContainerStarted","Data":"239e853a196c0e1ad34b5ea8d0e062dc9af22f29cebebd66eebe054bf1b2ef3e"} Jul 20 17:50:11.616285 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:11.616264 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gkzdq" event={"ID":"e8490308-78b2-45e7-b449-2126c3cd30a9","Type":"ContainerStarted","Data":"1cd76740416a2d5bbe578c1a1bf5be773c9c842e396ebdfebe1131df3034023f"} Jul 20 17:50:13.284161 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:13.284134 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:50:13.626940 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:13.626902 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg" event={"ID":"8a474b76-8147-4f5a-8bf6-933ca8943292","Type":"ContainerStarted","Data":"b6b1aba5e2732eef3d50d94a3e8a29c8f341bd1f2e30333619a08d8eb68eaf66"} Jul 20 17:50:13.627211 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:13.626968 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg" event={"ID":"8a474b76-8147-4f5a-8bf6-933ca8943292","Type":"ContainerStarted","Data":"1532d2a0694a6955904fa760f665f4670553e4549cc1b155893bc1335cd0d0c8"} Jul 20 17:50:13.650182 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:13.650130 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-7nzqg" podStartSLOduration=2.121565162 podStartE2EDuration="3.650111891s" totalImagesPullingTime="1.528546729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:50:13.650064649 +0000 UTC m=+122.021853222" watchObservedRunningTime="2026-07-20 17:50:13.650111891 +0000 UTC m=+122.021900464" Jul 20 17:50:15.635715 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:15.635631 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gkzdq" event={"ID":"e8490308-78b2-45e7-b449-2126c3cd30a9","Type":"ContainerStarted","Data":"c1536554c17f851fc50d86478b07e2aa89163c49a07e4689be25983950e042d2"} Jul 20 17:50:16.909343 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:16.909305 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb"] Jul 20 17:50:16.912332 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:16.912305 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" Jul 20 17:50:16.920770 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:16.920750 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 20 17:50:16.920881 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:16.920789 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cdfpv\"" Jul 20 17:50:16.929983 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:16.929956 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 20 17:50:16.967805 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:16.967777 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb"] Jul 20 17:50:17.001764 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.001706 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74fbff29-2608-4e88-a45f-dabff2fe558f-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-65rhb\" (UID: \"74fbff29-2608-4e88-a45f-dabff2fe558f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" Jul 20 17:50:17.001957 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.001806 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74fbff29-2608-4e88-a45f-dabff2fe558f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-65rhb\" (UID: \"74fbff29-2608-4e88-a45f-dabff2fe558f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" Jul 20 17:50:17.001957 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.001841 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6jp8\" (UniqueName: \"kubernetes.io/projected/74fbff29-2608-4e88-a45f-dabff2fe558f-kube-api-access-p6jp8\") pod \"openshift-state-metrics-79f6958cdf-65rhb\" (UID: \"74fbff29-2608-4e88-a45f-dabff2fe558f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" Jul 20 17:50:17.001957 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.001869 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74fbff29-2608-4e88-a45f-dabff2fe558f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-65rhb\" (UID: \"74fbff29-2608-4e88-a45f-dabff2fe558f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" Jul 20 17:50:17.102958 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.102928 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74fbff29-2608-4e88-a45f-dabff2fe558f-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-65rhb\" (UID: \"74fbff29-2608-4e88-a45f-dabff2fe558f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" Jul 20 17:50:17.103156 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.102976 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74fbff29-2608-4e88-a45f-dabff2fe558f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-65rhb\" (UID: \"74fbff29-2608-4e88-a45f-dabff2fe558f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" Jul 20 17:50:17.103156 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.103003 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6jp8\" (UniqueName: \"kubernetes.io/projected/74fbff29-2608-4e88-a45f-dabff2fe558f-kube-api-access-p6jp8\") pod \"openshift-state-metrics-79f6958cdf-65rhb\" (UID: \"74fbff29-2608-4e88-a45f-dabff2fe558f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" Jul 20 17:50:17.103156 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.103030 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74fbff29-2608-4e88-a45f-dabff2fe558f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-65rhb\" (UID: \"74fbff29-2608-4e88-a45f-dabff2fe558f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" Jul 20 17:50:17.103883 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.103861 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74fbff29-2608-4e88-a45f-dabff2fe558f-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-65rhb\" (UID: \"74fbff29-2608-4e88-a45f-dabff2fe558f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" Jul 20 17:50:17.105857 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.105835 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74fbff29-2608-4e88-a45f-dabff2fe558f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-65rhb\" (UID: \"74fbff29-2608-4e88-a45f-dabff2fe558f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" Jul 20 17:50:17.105960 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.105840 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74fbff29-2608-4e88-a45f-dabff2fe558f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-65rhb\" (UID: \"74fbff29-2608-4e88-a45f-dabff2fe558f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" Jul 20 17:50:17.233246 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.233217 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hbfq6"] Jul 20 17:50:17.236321 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.236300 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.252081 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.252058 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 20 17:50:17.252249 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.252228 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 20 17:50:17.252321 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.252309 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 20 17:50:17.374235 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.374192 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6jp8\" (UniqueName: \"kubernetes.io/projected/74fbff29-2608-4e88-a45f-dabff2fe558f-kube-api-access-p6jp8\") pod \"openshift-state-metrics-79f6958cdf-65rhb\" (UID: \"74fbff29-2608-4e88-a45f-dabff2fe558f\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" Jul 20 17:50:17.401225 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.401196 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2dmjq\"" Jul 20 17:50:17.406368 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.406326 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-metrics-client-ca\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.406540 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.406376 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-sys\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.406540 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.406407 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.406540 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.406440 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-node-exporter-tls\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.406540 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.406464 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-node-exporter-textfile\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.406768 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.406536 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-node-exporter-wtmp\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.406768 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.406603 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkqrw\" (UniqueName: \"kubernetes.io/projected/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-kube-api-access-kkqrw\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.406768 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.406657 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-root\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.406768 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.406701 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-node-exporter-accelerators-collector-config\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.508029 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.507937 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.508029 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.507980 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-node-exporter-tls\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.508029 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.508009 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-node-exporter-textfile\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.508308 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.508043 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-node-exporter-wtmp\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.508308 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.508091 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkqrw\" (UniqueName: \"kubernetes.io/projected/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-kube-api-access-kkqrw\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.508308 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.508128 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-root\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.508308 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.508168 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-node-exporter-accelerators-collector-config\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.508308 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.508223 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-metrics-client-ca\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.508308 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.508245 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-node-exporter-wtmp\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.508308 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.508254 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-sys\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.508308 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.508298 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-sys\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.508689 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.508350 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-root\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.508689 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.508421 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-node-exporter-textfile\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.508910 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.508864 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-node-exporter-accelerators-collector-config\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.508910 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.508896 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-metrics-client-ca\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.510558 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.510535 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.510662 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.510653 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-node-exporter-tls\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.521528 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.521505 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" Jul 20 17:50:17.583118 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.583086 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkqrw\" (UniqueName: \"kubernetes.io/projected/d242fdc9-a9a1-4064-af05-7e2dee5d87c5-kube-api-access-kkqrw\") pod \"node-exporter-hbfq6\" (UID: \"d242fdc9-a9a1-4064-af05-7e2dee5d87c5\") " pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:17.845875 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:17.845799 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hbfq6" Jul 20 17:50:18.976716 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:50:18.976365 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd242fdc9_a9a1_4064_af05_7e2dee5d87c5.slice/crio-52aa3ed83da358ed1f83fe41948150a4d5c20c5498e278151b66cf49ab911a1b WatchSource:0}: Error finding container 52aa3ed83da358ed1f83fe41948150a4d5c20c5498e278151b66cf49ab911a1b: Status 404 returned error can't find the container with id 52aa3ed83da358ed1f83fe41948150a4d5c20c5498e278151b66cf49ab911a1b Jul 20 17:50:19.127863 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:19.127833 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb"] Jul 20 17:50:19.131582 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:50:19.131558 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fbff29_2608_4e88_a45f_dabff2fe558f.slice/crio-ac184ce1502ac60a8dddc8fd1af10d8d91c951c7bc26e6002a02d2b4a1a6d1f3 WatchSource:0}: Error finding container ac184ce1502ac60a8dddc8fd1af10d8d91c951c7bc26e6002a02d2b4a1a6d1f3: Status 404 returned error can't find the container with id ac184ce1502ac60a8dddc8fd1af10d8d91c951c7bc26e6002a02d2b4a1a6d1f3 Jul 20 17:50:19.651824 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:19.651707 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hbfq6" event={"ID":"d242fdc9-a9a1-4064-af05-7e2dee5d87c5","Type":"ContainerStarted","Data":"52aa3ed83da358ed1f83fe41948150a4d5c20c5498e278151b66cf49ab911a1b"} Jul 20 17:50:19.653813 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:19.653786 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gkzdq" event={"ID":"e8490308-78b2-45e7-b449-2126c3cd30a9","Type":"ContainerStarted","Data":"f160b83ffc56e819f635ead625b17018ab66e9337ed1798abb1f2bf8df8c580b"} Jul 20 17:50:19.655481 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:19.655461 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" event={"ID":"74fbff29-2608-4e88-a45f-dabff2fe558f","Type":"ContainerStarted","Data":"b050b4d37090fcaa3d9aeb305cac3d15e677c9b0edf00a1bfaf3ead4b059a92e"} Jul 20 17:50:19.655481 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:19.655501 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" event={"ID":"74fbff29-2608-4e88-a45f-dabff2fe558f","Type":"ContainerStarted","Data":"5ed06a301695c6b36145c8945b07ddafedc03a09371062715a5751173e0a03a0"} Jul 20 17:50:19.655633 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:19.655519 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" event={"ID":"74fbff29-2608-4e88-a45f-dabff2fe558f","Type":"ContainerStarted","Data":"ac184ce1502ac60a8dddc8fd1af10d8d91c951c7bc26e6002a02d2b4a1a6d1f3"} Jul 20 17:50:19.700985 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:19.700928 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gkzdq" podStartSLOduration=33.217554458 podStartE2EDuration="45.700907234s" totalImagesPullingTime="12.483352776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:49:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:50:19.700475482 +0000 UTC m=+128.072264056" watchObservedRunningTime="2026-07-20 17:50:19.700907234 +0000 UTC m=+128.072695809" Jul 20 17:50:20.661794 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:20.661762 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" event={"ID":"74fbff29-2608-4e88-a45f-dabff2fe558f","Type":"ContainerStarted","Data":"41dc0adeb33a2e3a71beeab3dc9fb81a2629ce4e0b3ffcb77eb137daf708a160"} Jul 20 17:50:20.663304 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:20.663283 2583 generic.go:356] "Generic (PLEG): container finished" podID="d242fdc9-a9a1-4064-af05-7e2dee5d87c5" containerID="c0b44f81b02a373df3654d53f180240d9826bdabdf14bfaf700628bb17ec6631" exitCode=0 Jul 20 17:50:20.663424 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:20.663371 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hbfq6" event={"ID":"d242fdc9-a9a1-4064-af05-7e2dee5d87c5","Type":"ContainerDied","Data":"c0b44f81b02a373df3654d53f180240d9826bdabdf14bfaf700628bb17ec6631"} Jul 20 17:50:20.751167 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:20.751122 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-65rhb" podStartSLOduration=3.489262833 podStartE2EDuration="4.751106827s" totalImagesPullingTime="1.261843994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:50:20.750232295 +0000 UTC m=+129.122020882" watchObservedRunningTime="2026-07-20 17:50:20.751106827 +0000 UTC m=+129.122895437" Jul 20 17:50:21.668338 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:21.668297 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hbfq6" event={"ID":"d242fdc9-a9a1-4064-af05-7e2dee5d87c5","Type":"ContainerStarted","Data":"eccd7f195827f2e07342e716d6f3bf0539c832cd553be309116062537783ae0e"} Jul 20 17:50:21.668787 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:21.668364 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hbfq6" event={"ID":"d242fdc9-a9a1-4064-af05-7e2dee5d87c5","Type":"ContainerStarted","Data":"3e767fcdb24db6c8bda1585428132191baadb0a514f0cce08095f2672e4f5ad4"} Jul 20 17:50:21.709852 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:21.709804 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hbfq6" podStartSLOduration=3.629007161 podStartE2EDuration="4.709789734s" totalImagesPullingTime="1.013620832s" totalInitContainerRuntime="67.161741ms" isStatefulPod=true podCreationTimestamp="2026-07-20 17:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:50:21.709564592 +0000 UTC m=+130.081353170" watchObservedRunningTime="2026-07-20 17:50:21.709789734 +0000 UTC m=+130.081578308" Jul 20 17:50:21.846718 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:21.846686 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs\") pod \"network-metrics-daemon-r24gs\" (UID: \"76af6cb3-7b9d-4993-a5c8-24afeb42b60b\") " pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:50:21.849024 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:21.848993 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76af6cb3-7b9d-4993-a5c8-24afeb42b60b-metrics-certs\") pod \"network-metrics-daemon-r24gs\" (UID: \"76af6cb3-7b9d-4993-a5c8-24afeb42b60b\") " pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:50:21.994157 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:21.994129 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mjfj5\"" Jul 20 17:50:21.999448 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:21.999429 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r24gs" Jul 20 17:50:22.152910 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:22.152875 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r24gs"] Jul 20 17:50:22.155849 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:50:22.155816 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76af6cb3_7b9d_4993_a5c8_24afeb42b60b.slice/crio-1873871f3d14659d07a18150bf71cf420c395a5858697c95bc58c67e5143ad6f WatchSource:0}: Error finding container 1873871f3d14659d07a18150bf71cf420c395a5858697c95bc58c67e5143ad6f: Status 404 returned error can't find the container with id 1873871f3d14659d07a18150bf71cf420c395a5858697c95bc58c67e5143ad6f Jul 20 17:50:22.672864 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:22.672823 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r24gs" event={"ID":"76af6cb3-7b9d-4993-a5c8-24afeb42b60b","Type":"ContainerStarted","Data":"1873871f3d14659d07a18150bf71cf420c395a5858697c95bc58c67e5143ad6f"} Jul 20 17:50:23.678627 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:23.678540 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r24gs" event={"ID":"76af6cb3-7b9d-4993-a5c8-24afeb42b60b","Type":"ContainerStarted","Data":"43ec1dc3bb15c7c82c7b53860dfde2fea4b84d072c5a6801b0cdb82e397b308e"} Jul 20 17:50:23.678627 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:23.678591 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r24gs" event={"ID":"76af6cb3-7b9d-4993-a5c8-24afeb42b60b","Type":"ContainerStarted","Data":"5787a186accdbc9485f2792f29c470790d8736066b81cae0e1c0ad01045f99d1"} Jul 20 17:50:23.701696 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:23.701649 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r24gs" podStartSLOduration=130.573947579 podStartE2EDuration="2m11.701635494s" totalImagesPullingTime="1.127687915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:50:23.70128807 +0000 UTC m=+132.073076643" watchObservedRunningTime="2026-07-20 17:50:23.701635494 +0000 UTC m=+132.073424067" Jul 20 17:50:25.596973 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:25.596946 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cdfcc7df9-f7lqq" Jul 20 17:50:28.300363 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.300303 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" podUID="a69e79ec-082f-47fc-ad1b-d64aa66c7e3f" containerName="registry" containerID="cri-o://7140176b33b0063162304e3e5cb53373fe4aebb6426e6a0f9e2664f08b7bb3ed" gracePeriod=30 Jul 20 17:50:28.559234 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.559180 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:50:28.698188 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.698160 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-installation-pull-secrets\") pod \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " Jul 20 17:50:28.698366 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.698198 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-ca-trust-extracted\") pod \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " Jul 20 17:50:28.698366 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.698222 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-bound-sa-token\") pod \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " Jul 20 17:50:28.698366 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.698261 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-image-registry-private-configuration\") pod \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " Jul 20 17:50:28.698366 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.698297 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n47zd\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-kube-api-access-n47zd\") pod \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " Jul 20 17:50:28.698366 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.698345 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls\") pod \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " Jul 20 17:50:28.698632 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.698392 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-certificates\") pod \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " Jul 20 17:50:28.698632 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.698416 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-trusted-ca\") pod \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\" (UID: \"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f\") " Jul 20 17:50:28.699255 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.698965 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f" (UID: "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 17:50:28.699255 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.699051 2583 generic.go:356] "Generic (PLEG): container finished" podID="a69e79ec-082f-47fc-ad1b-d64aa66c7e3f" containerID="7140176b33b0063162304e3e5cb53373fe4aebb6426e6a0f9e2664f08b7bb3ed" exitCode=0 Jul 20 17:50:28.699255 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.699173 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" event={"ID":"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f","Type":"ContainerDied","Data":"7140176b33b0063162304e3e5cb53373fe4aebb6426e6a0f9e2664f08b7bb3ed"} Jul 20 17:50:28.699255 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.699231 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" event={"ID":"a69e79ec-082f-47fc-ad1b-d64aa66c7e3f","Type":"ContainerDied","Data":"cd5dc95135ef965802573463f513421dced55414e95368e2a761b67672d7480e"} Jul 20 17:50:28.699544 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.699260 2583 scope.go:117] "RemoveContainer" containerID="7140176b33b0063162304e3e5cb53373fe4aebb6426e6a0f9e2664f08b7bb3ed" Jul 20 17:50:28.699544 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.699444 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97ccd8d99-4ckws" Jul 20 17:50:28.699910 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.699871 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f" (UID: "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 17:50:28.701037 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.700999 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f" (UID: "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 17:50:28.701156 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.701134 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f" (UID: "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 17:50:28.701286 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.701259 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f" (UID: "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 17:50:28.701372 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.701354 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f" (UID: "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 17:50:28.701550 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.701524 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-kube-api-access-n47zd" (OuterVolumeSpecName: "kube-api-access-n47zd") pod "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f" (UID: "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f"). InnerVolumeSpecName "kube-api-access-n47zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 17:50:28.708239 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.708213 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f" (UID: "a69e79ec-082f-47fc-ad1b-d64aa66c7e3f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 17:50:28.716198 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.716180 2583 scope.go:117] "RemoveContainer" containerID="7140176b33b0063162304e3e5cb53373fe4aebb6426e6a0f9e2664f08b7bb3ed" Jul 20 17:50:28.716458 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:50:28.716436 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7140176b33b0063162304e3e5cb53373fe4aebb6426e6a0f9e2664f08b7bb3ed\": container with ID starting with 7140176b33b0063162304e3e5cb53373fe4aebb6426e6a0f9e2664f08b7bb3ed not found: ID does not exist" containerID="7140176b33b0063162304e3e5cb53373fe4aebb6426e6a0f9e2664f08b7bb3ed" Jul 20 17:50:28.716525 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.716470 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7140176b33b0063162304e3e5cb53373fe4aebb6426e6a0f9e2664f08b7bb3ed"} err="failed to get container status \"7140176b33b0063162304e3e5cb53373fe4aebb6426e6a0f9e2664f08b7bb3ed\": rpc error: code = NotFound desc = could not find container \"7140176b33b0063162304e3e5cb53373fe4aebb6426e6a0f9e2664f08b7bb3ed\": container with ID starting with 7140176b33b0063162304e3e5cb53373fe4aebb6426e6a0f9e2664f08b7bb3ed not found: ID does not exist" Jul 20 17:50:28.799894 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.799855 2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-tls\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 17:50:28.799894 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.799892 2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-registry-certificates\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 17:50:28.799894 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.799902 2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-trusted-ca\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 17:50:28.800076 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.799912 2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-installation-pull-secrets\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 17:50:28.800076 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.799920 2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-ca-trust-extracted\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 17:50:28.800076 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.799929 2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-bound-sa-token\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 17:50:28.800076 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.799938 2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-image-registry-private-configuration\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 17:50:28.800076 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:28.799946 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n47zd\" (UniqueName: \"kubernetes.io/projected/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f-kube-api-access-n47zd\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 17:50:29.021058 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:29.021025 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-97ccd8d99-4ckws"] Jul 20 17:50:29.024838 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:29.024817 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-97ccd8d99-4ckws"] Jul 20 17:50:30.170486 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:30.170453 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a69e79ec-082f-47fc-ad1b-d64aa66c7e3f" path="/var/lib/kubelet/pods/a69e79ec-082f-47fc-ad1b-d64aa66c7e3f/volumes" Jul 20 17:50:44.759040 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:44.758961 2583 generic.go:356] "Generic (PLEG): container finished" podID="6f897e0b-72f4-40a1-917e-8e03196cedfe" containerID="3840df42a4a0d353262256cca0bbe6b886159100537aa116d98326df6c452303" exitCode=0 Jul 20 17:50:44.759382 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:44.759033 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9" event={"ID":"6f897e0b-72f4-40a1-917e-8e03196cedfe","Type":"ContainerDied","Data":"3840df42a4a0d353262256cca0bbe6b886159100537aa116d98326df6c452303"} Jul 20 17:50:44.759382 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:44.759338 2583 scope.go:117] "RemoveContainer" containerID="3840df42a4a0d353262256cca0bbe6b886159100537aa116d98326df6c452303" Jul 20 17:50:45.765613 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:45.765584 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xrcw9" event={"ID":"6f897e0b-72f4-40a1-917e-8e03196cedfe","Type":"ContainerStarted","Data":"ee879e427b65ba0da0251c33db856258a76258b679ac1e4554c38ee81c60533f"} Jul 20 17:50:54.797513 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:54.797484 2583 generic.go:356] "Generic (PLEG): container finished" podID="7cc112b9-237b-4b0b-b009-a2416a7a8039" containerID="7d445d4cb25d059faf32b5fe303d5b7416908adca51e76a7059d91f658cf572f" exitCode=0 Jul 20 17:50:54.797919 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:54.797548 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc" event={"ID":"7cc112b9-237b-4b0b-b009-a2416a7a8039","Type":"ContainerDied","Data":"7d445d4cb25d059faf32b5fe303d5b7416908adca51e76a7059d91f658cf572f"} Jul 20 17:50:54.797919 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:54.797868 2583 scope.go:117] "RemoveContainer" containerID="7d445d4cb25d059faf32b5fe303d5b7416908adca51e76a7059d91f658cf572f" Jul 20 17:50:55.802592 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:50:55.802560 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-jslvc" event={"ID":"7cc112b9-237b-4b0b-b009-a2416a7a8039","Type":"ContainerStarted","Data":"df56a98b5c7ead4b45b9254ad8693ddb2bba8f242d80a26e2e522d5e7bc44aca"} Jul 20 17:53:11.979031 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:53:11.978990 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 17:53:11.984425 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:53:11.984404 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 17:53:11.989548 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:53:11.989524 2583 kubelet.go:1648] "Image garbage collection succeeded" Jul 20 17:57:42.731925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:42.731890 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl"] Jul 20 17:57:42.732417 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:42.732184 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a69e79ec-082f-47fc-ad1b-d64aa66c7e3f" containerName="registry" Jul 20 17:57:42.732417 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:42.732196 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a69e79ec-082f-47fc-ad1b-d64aa66c7e3f" containerName="registry" Jul 20 17:57:42.732417 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:42.732265 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a69e79ec-082f-47fc-ad1b-d64aa66c7e3f" containerName="registry" Jul 20 17:57:42.735103 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:42.735089 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl" Jul 20 17:57:42.737653 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:42.737634 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 20 17:57:42.737782 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:42.737658 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-t695n\"" Jul 20 17:57:42.738507 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:42.738488 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 20 17:57:42.743596 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:42.743576 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl"] Jul 20 17:57:42.892613 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:42.892577 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swsgp\" (UniqueName: \"kubernetes.io/projected/85f5613e-4260-4512-a27d-0ea0aee29dc9-kube-api-access-swsgp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl\" (UID: \"85f5613e-4260-4512-a27d-0ea0aee29dc9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl" Jul 20 17:57:42.892613 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:42.892619 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85f5613e-4260-4512-a27d-0ea0aee29dc9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl\" (UID: \"85f5613e-4260-4512-a27d-0ea0aee29dc9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl" Jul 20 17:57:42.892887 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:42.892661 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85f5613e-4260-4512-a27d-0ea0aee29dc9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl\" (UID: \"85f5613e-4260-4512-a27d-0ea0aee29dc9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl" Jul 20 17:57:42.993288 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:42.993184 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85f5613e-4260-4512-a27d-0ea0aee29dc9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl\" (UID: \"85f5613e-4260-4512-a27d-0ea0aee29dc9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl" Jul 20 17:57:42.993288 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:42.993274 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swsgp\" (UniqueName: \"kubernetes.io/projected/85f5613e-4260-4512-a27d-0ea0aee29dc9-kube-api-access-swsgp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl\" (UID: \"85f5613e-4260-4512-a27d-0ea0aee29dc9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl" Jul 20 17:57:42.993497 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:42.993305 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85f5613e-4260-4512-a27d-0ea0aee29dc9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl\" (UID: \"85f5613e-4260-4512-a27d-0ea0aee29dc9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl" Jul 20 17:57:42.993583 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:42.993562 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85f5613e-4260-4512-a27d-0ea0aee29dc9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl\" (UID: \"85f5613e-4260-4512-a27d-0ea0aee29dc9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl" Jul 20 17:57:42.993674 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:42.993629 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85f5613e-4260-4512-a27d-0ea0aee29dc9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl\" (UID: \"85f5613e-4260-4512-a27d-0ea0aee29dc9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl" Jul 20 17:57:43.001439 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:43.001417 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swsgp\" (UniqueName: \"kubernetes.io/projected/85f5613e-4260-4512-a27d-0ea0aee29dc9-kube-api-access-swsgp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl\" (UID: \"85f5613e-4260-4512-a27d-0ea0aee29dc9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl" Jul 20 17:57:43.043870 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:43.043838 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl" Jul 20 17:57:43.172409 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:43.172378 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl"] Jul 20 17:57:43.176796 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:57:43.176770 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f5613e_4260_4512_a27d_0ea0aee29dc9.slice/crio-422cae410c9ebdb661dad064869fc537273ce8ed51681879e153242c3f46c73e WatchSource:0}: Error finding container 422cae410c9ebdb661dad064869fc537273ce8ed51681879e153242c3f46c73e: Status 404 returned error can't find the container with id 422cae410c9ebdb661dad064869fc537273ce8ed51681879e153242c3f46c73e Jul 20 17:57:43.178779 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:43.178760 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 17:57:43.201604 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:43.201577 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl" event={"ID":"85f5613e-4260-4512-a27d-0ea0aee29dc9","Type":"ContainerStarted","Data":"422cae410c9ebdb661dad064869fc537273ce8ed51681879e153242c3f46c73e"} Jul 20 17:57:51.235967 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:51.235927 2583 generic.go:356] "Generic (PLEG): container finished" podID="85f5613e-4260-4512-a27d-0ea0aee29dc9" containerID="f87e986310a3d3011cf986c44d2d03585b3c12a1f0fff237f92e6366318f6395" exitCode=0 Jul 20 17:57:51.236378 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:51.235976 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl" event={"ID":"85f5613e-4260-4512-a27d-0ea0aee29dc9","Type":"ContainerDied","Data":"f87e986310a3d3011cf986c44d2d03585b3c12a1f0fff237f92e6366318f6395"} Jul 20 17:57:53.246123 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:53.246034 2583 generic.go:356] "Generic (PLEG): container finished" podID="85f5613e-4260-4512-a27d-0ea0aee29dc9" containerID="a4c3ae9a95147248cdf9232100d9aaf0e6f7bde2c2b9f649ad9a46564c7eb44a" exitCode=0 Jul 20 17:57:53.246123 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:57:53.246094 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl" event={"ID":"85f5613e-4260-4512-a27d-0ea0aee29dc9","Type":"ContainerDied","Data":"a4c3ae9a95147248cdf9232100d9aaf0e6f7bde2c2b9f649ad9a46564c7eb44a"} Jul 20 17:58:01.280555 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:01.280474 2583 generic.go:356] "Generic (PLEG): container finished" podID="85f5613e-4260-4512-a27d-0ea0aee29dc9" containerID="b2ff427f2ebc5c1080213d6e578c242c3ff58be67366044465fb505fd6343042" exitCode=0 Jul 20 17:58:01.280555 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:01.280540 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl" event={"ID":"85f5613e-4260-4512-a27d-0ea0aee29dc9","Type":"ContainerDied","Data":"b2ff427f2ebc5c1080213d6e578c242c3ff58be67366044465fb505fd6343042"} Jul 20 17:58:02.405656 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:02.405633 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl" Jul 20 17:58:02.442261 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:02.442226 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swsgp\" (UniqueName: \"kubernetes.io/projected/85f5613e-4260-4512-a27d-0ea0aee29dc9-kube-api-access-swsgp\") pod \"85f5613e-4260-4512-a27d-0ea0aee29dc9\" (UID: \"85f5613e-4260-4512-a27d-0ea0aee29dc9\") " Jul 20 17:58:02.442261 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:02.442261 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85f5613e-4260-4512-a27d-0ea0aee29dc9-util\") pod \"85f5613e-4260-4512-a27d-0ea0aee29dc9\" (UID: \"85f5613e-4260-4512-a27d-0ea0aee29dc9\") " Jul 20 17:58:02.442486 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:02.442303 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85f5613e-4260-4512-a27d-0ea0aee29dc9-bundle\") pod \"85f5613e-4260-4512-a27d-0ea0aee29dc9\" (UID: \"85f5613e-4260-4512-a27d-0ea0aee29dc9\") " Jul 20 17:58:02.443026 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:02.443002 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f5613e-4260-4512-a27d-0ea0aee29dc9-bundle" (OuterVolumeSpecName: "bundle") pod "85f5613e-4260-4512-a27d-0ea0aee29dc9" (UID: "85f5613e-4260-4512-a27d-0ea0aee29dc9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 17:58:02.444429 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:02.444401 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f5613e-4260-4512-a27d-0ea0aee29dc9-kube-api-access-swsgp" (OuterVolumeSpecName: "kube-api-access-swsgp") pod "85f5613e-4260-4512-a27d-0ea0aee29dc9" (UID: "85f5613e-4260-4512-a27d-0ea0aee29dc9"). InnerVolumeSpecName "kube-api-access-swsgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 17:58:02.446645 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:02.446628 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f5613e-4260-4512-a27d-0ea0aee29dc9-util" (OuterVolumeSpecName: "util") pod "85f5613e-4260-4512-a27d-0ea0aee29dc9" (UID: "85f5613e-4260-4512-a27d-0ea0aee29dc9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 17:58:02.543066 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:02.542982 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swsgp\" (UniqueName: \"kubernetes.io/projected/85f5613e-4260-4512-a27d-0ea0aee29dc9-kube-api-access-swsgp\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 17:58:02.543066 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:02.543014 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85f5613e-4260-4512-a27d-0ea0aee29dc9-util\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 17:58:02.543066 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:02.543024 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85f5613e-4260-4512-a27d-0ea0aee29dc9-bundle\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 17:58:03.288580 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:03.288552 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl" Jul 20 17:58:03.288769 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:03.288551 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phjcl" event={"ID":"85f5613e-4260-4512-a27d-0ea0aee29dc9","Type":"ContainerDied","Data":"422cae410c9ebdb661dad064869fc537273ce8ed51681879e153242c3f46c73e"} Jul 20 17:58:03.288769 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:03.288668 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="422cae410c9ebdb661dad064869fc537273ce8ed51681879e153242c3f46c73e" Jul 20 17:58:10.937939 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:10.937907 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sq8bn"] Jul 20 17:58:10.938413 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:10.938167 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f5613e-4260-4512-a27d-0ea0aee29dc9" containerName="extract" Jul 20 17:58:10.938413 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:10.938176 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f5613e-4260-4512-a27d-0ea0aee29dc9" containerName="extract" Jul 20 17:58:10.938413 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:10.938187 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f5613e-4260-4512-a27d-0ea0aee29dc9" containerName="pull" Jul 20 17:58:10.938413 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:10.938192 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f5613e-4260-4512-a27d-0ea0aee29dc9" containerName="pull" Jul 20 17:58:10.938413 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:10.938209 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f5613e-4260-4512-a27d-0ea0aee29dc9" containerName="util" Jul 20 17:58:10.938413 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:10.938215 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f5613e-4260-4512-a27d-0ea0aee29dc9" containerName="util" Jul 20 17:58:10.938413 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:10.938257 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="85f5613e-4260-4512-a27d-0ea0aee29dc9" containerName="extract" Jul 20 17:58:10.942005 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:10.941988 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sq8bn" Jul 20 17:58:10.944810 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:10.944792 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 20 17:58:10.945274 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:10.945259 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-nljh2\"" Jul 20 17:58:10.945421 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:10.945405 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 20 17:58:10.955123 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:10.955080 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sq8bn"] Jul 20 17:58:11.007984 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:11.007951 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/355680c9-f7f6-4504-838b-78c67d8db77c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sq8bn\" (UID: \"355680c9-f7f6-4504-838b-78c67d8db77c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sq8bn" Jul 20 17:58:11.007984 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:11.007988 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gkq4\" (UniqueName: \"kubernetes.io/projected/355680c9-f7f6-4504-838b-78c67d8db77c-kube-api-access-4gkq4\") pod \"cert-manager-operator-controller-manager-55c444bf86-sq8bn\" (UID: \"355680c9-f7f6-4504-838b-78c67d8db77c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sq8bn" Jul 20 17:58:11.108982 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:11.108950 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/355680c9-f7f6-4504-838b-78c67d8db77c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sq8bn\" (UID: \"355680c9-f7f6-4504-838b-78c67d8db77c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sq8bn" Jul 20 17:58:11.109128 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:11.108989 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gkq4\" (UniqueName: \"kubernetes.io/projected/355680c9-f7f6-4504-838b-78c67d8db77c-kube-api-access-4gkq4\") pod \"cert-manager-operator-controller-manager-55c444bf86-sq8bn\" (UID: \"355680c9-f7f6-4504-838b-78c67d8db77c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sq8bn" Jul 20 17:58:11.109333 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:11.109313 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/355680c9-f7f6-4504-838b-78c67d8db77c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sq8bn\" (UID: \"355680c9-f7f6-4504-838b-78c67d8db77c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sq8bn" Jul 20 17:58:11.120778 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:11.120755 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gkq4\" (UniqueName: \"kubernetes.io/projected/355680c9-f7f6-4504-838b-78c67d8db77c-kube-api-access-4gkq4\") pod \"cert-manager-operator-controller-manager-55c444bf86-sq8bn\" (UID: \"355680c9-f7f6-4504-838b-78c67d8db77c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sq8bn" Jul 20 17:58:11.255481 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:11.255395 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sq8bn" Jul 20 17:58:11.398805 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:11.398762 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sq8bn"] Jul 20 17:58:11.402468 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:58:11.402443 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355680c9_f7f6_4504_838b_78c67d8db77c.slice/crio-75c01974a66163ad235593b7652be8ec3cecb3c17fc83c13880665dbda688a1c WatchSource:0}: Error finding container 75c01974a66163ad235593b7652be8ec3cecb3c17fc83c13880665dbda688a1c: Status 404 returned error can't find the container with id 75c01974a66163ad235593b7652be8ec3cecb3c17fc83c13880665dbda688a1c Jul 20 17:58:12.005229 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:12.005199 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 17:58:12.006854 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:12.006837 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 17:58:12.320909 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:12.320827 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sq8bn" event={"ID":"355680c9-f7f6-4504-838b-78c67d8db77c","Type":"ContainerStarted","Data":"75c01974a66163ad235593b7652be8ec3cecb3c17fc83c13880665dbda688a1c"} Jul 20 17:58:15.334940 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:15.334893 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sq8bn" event={"ID":"355680c9-f7f6-4504-838b-78c67d8db77c","Type":"ContainerStarted","Data":"f73af27d4e9b0ed15d1868a9fe8b481729b0790adab18b5c27b1f9d5e886025b"} Jul 20 17:58:15.359395 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:15.359348 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sq8bn" podStartSLOduration=1.513511742 podStartE2EDuration="5.359332012s" totalImagesPullingTime="3.84582027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:58:15.358223733 +0000 UTC m=+603.730012309" watchObservedRunningTime="2026-07-20 17:58:15.359332012 +0000 UTC m=+603.731120586" Jul 20 17:58:16.012990 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.012956 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs"] Jul 20 17:58:16.015145 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.015113 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs" Jul 20 17:58:16.018067 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.018047 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-t695n\"" Jul 20 17:58:16.018383 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.018365 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 20 17:58:16.018465 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.018372 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 20 17:58:16.045446 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.045418 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs"] Jul 20 17:58:16.143834 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.143791 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6ee24b4-538a-4e89-abfb-b99f7dde3b9c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs\" (UID: \"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs" Jul 20 17:58:16.144004 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.143902 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khwm4\" (UniqueName: \"kubernetes.io/projected/d6ee24b4-538a-4e89-abfb-b99f7dde3b9c-kube-api-access-khwm4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs\" (UID: \"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs" Jul 20 17:58:16.144004 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.143967 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6ee24b4-538a-4e89-abfb-b99f7dde3b9c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs\" (UID: \"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs" Jul 20 17:58:16.245035 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.244997 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6ee24b4-538a-4e89-abfb-b99f7dde3b9c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs\" (UID: \"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs" Jul 20 17:58:16.245220 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.245071 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6ee24b4-538a-4e89-abfb-b99f7dde3b9c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs\" (UID: \"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs" Jul 20 17:58:16.245220 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.245115 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khwm4\" (UniqueName: \"kubernetes.io/projected/d6ee24b4-538a-4e89-abfb-b99f7dde3b9c-kube-api-access-khwm4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs\" (UID: \"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs" Jul 20 17:58:16.245476 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.245443 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6ee24b4-538a-4e89-abfb-b99f7dde3b9c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs\" (UID: \"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs" Jul 20 17:58:16.245476 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.245467 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6ee24b4-538a-4e89-abfb-b99f7dde3b9c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs\" (UID: \"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs" Jul 20 17:58:16.278833 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.278754 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khwm4\" (UniqueName: \"kubernetes.io/projected/d6ee24b4-538a-4e89-abfb-b99f7dde3b9c-kube-api-access-khwm4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs\" (UID: \"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs" Jul 20 17:58:16.324561 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.324531 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs" Jul 20 17:58:16.537769 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.537677 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs"] Jul 20 17:58:16.540936 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:58:16.540905 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ee24b4_538a_4e89_abfb_b99f7dde3b9c.slice/crio-ad6653f40db4cf4428d7f5b49cd4f8cc280d70597a1c2159c26c9e07a6ff8986 WatchSource:0}: Error finding container ad6653f40db4cf4428d7f5b49cd4f8cc280d70597a1c2159c26c9e07a6ff8986: Status 404 returned error can't find the container with id ad6653f40db4cf4428d7f5b49cd4f8cc280d70597a1c2159c26c9e07a6ff8986 Jul 20 17:58:16.869332 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.869245 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ptjpn"] Jul 20 17:58:16.871189 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.871173 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-ptjpn" Jul 20 17:58:16.874177 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.874155 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 20 17:58:16.874312 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.874234 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 20 17:58:16.874312 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.874260 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-5zc9v\"" Jul 20 17:58:16.884195 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.884176 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ptjpn"] Jul 20 17:58:16.951514 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.951476 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ce2898c-dda5-44f2-8a60-0f742e434807-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ptjpn\" (UID: \"2ce2898c-dda5-44f2-8a60-0f742e434807\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ptjpn" Jul 20 17:58:16.951696 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:16.951559 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9trb\" (UniqueName: \"kubernetes.io/projected/2ce2898c-dda5-44f2-8a60-0f742e434807-kube-api-access-j9trb\") pod \"cert-manager-webhook-6486ffd997-ptjpn\" (UID: \"2ce2898c-dda5-44f2-8a60-0f742e434807\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ptjpn" Jul 20 17:58:17.051993 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:17.051953 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9trb\" (UniqueName: \"kubernetes.io/projected/2ce2898c-dda5-44f2-8a60-0f742e434807-kube-api-access-j9trb\") pod \"cert-manager-webhook-6486ffd997-ptjpn\" (UID: \"2ce2898c-dda5-44f2-8a60-0f742e434807\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ptjpn" Jul 20 17:58:17.051993 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:17.051993 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ce2898c-dda5-44f2-8a60-0f742e434807-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ptjpn\" (UID: \"2ce2898c-dda5-44f2-8a60-0f742e434807\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ptjpn" Jul 20 17:58:17.083938 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:17.083913 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9trb\" (UniqueName: \"kubernetes.io/projected/2ce2898c-dda5-44f2-8a60-0f742e434807-kube-api-access-j9trb\") pod \"cert-manager-webhook-6486ffd997-ptjpn\" (UID: \"2ce2898c-dda5-44f2-8a60-0f742e434807\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ptjpn" Jul 20 17:58:17.084271 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:17.084251 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ce2898c-dda5-44f2-8a60-0f742e434807-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ptjpn\" (UID: \"2ce2898c-dda5-44f2-8a60-0f742e434807\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ptjpn" Jul 20 17:58:17.191249 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:17.191220 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-ptjpn" Jul 20 17:58:17.344172 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:17.344141 2583 generic.go:356] "Generic (PLEG): container finished" podID="d6ee24b4-538a-4e89-abfb-b99f7dde3b9c" containerID="9b8f49b2329e8dda87da9e1d927b41913d699ae6c2a419500e79ab89c79fb310" exitCode=0 Jul 20 17:58:17.344317 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:17.344182 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs" event={"ID":"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c","Type":"ContainerDied","Data":"9b8f49b2329e8dda87da9e1d927b41913d699ae6c2a419500e79ab89c79fb310"} Jul 20 17:58:17.344317 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:17.344225 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs" event={"ID":"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c","Type":"ContainerStarted","Data":"ad6653f40db4cf4428d7f5b49cd4f8cc280d70597a1c2159c26c9e07a6ff8986"} Jul 20 17:58:17.347880 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:17.347859 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ptjpn"] Jul 20 17:58:17.351066 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:58:17.351044 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ce2898c_dda5_44f2_8a60_0f742e434807.slice/crio-bfcdee8c992e818195c5ebc0a76f5d66841cbc40796dfb2eb173808750b47249 WatchSource:0}: Error finding container bfcdee8c992e818195c5ebc0a76f5d66841cbc40796dfb2eb173808750b47249: Status 404 returned error can't find the container with id bfcdee8c992e818195c5ebc0a76f5d66841cbc40796dfb2eb173808750b47249 Jul 20 17:58:18.353503 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:18.353442 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-ptjpn" event={"ID":"2ce2898c-dda5-44f2-8a60-0f742e434807","Type":"ContainerStarted","Data":"bfcdee8c992e818195c5ebc0a76f5d66841cbc40796dfb2eb173808750b47249"} Jul 20 17:58:19.994245 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:19.994206 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4g56j"] Jul 20 17:58:19.996581 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:19.996561 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4g56j" Jul 20 17:58:19.999858 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:19.999834 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5bgxb\"" Jul 20 17:58:20.010523 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:20.010504 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4g56j"] Jul 20 17:58:20.080167 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:20.080129 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6x7l\" (UniqueName: \"kubernetes.io/projected/1d474c1c-0534-4149-b026-e32b23029845-kube-api-access-d6x7l\") pod \"cert-manager-cainjector-559cdd8c87-4g56j\" (UID: \"1d474c1c-0534-4149-b026-e32b23029845\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4g56j" Jul 20 17:58:20.080344 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:20.080294 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d474c1c-0534-4149-b026-e32b23029845-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4g56j\" (UID: \"1d474c1c-0534-4149-b026-e32b23029845\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4g56j" Jul 20 17:58:20.181308 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:20.181273 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6x7l\" (UniqueName: \"kubernetes.io/projected/1d474c1c-0534-4149-b026-e32b23029845-kube-api-access-d6x7l\") pod \"cert-manager-cainjector-559cdd8c87-4g56j\" (UID: \"1d474c1c-0534-4149-b026-e32b23029845\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4g56j" Jul 20 17:58:20.181481 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:20.181366 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d474c1c-0534-4149-b026-e32b23029845-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4g56j\" (UID: \"1d474c1c-0534-4149-b026-e32b23029845\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4g56j" Jul 20 17:58:20.191865 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:20.191822 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6x7l\" (UniqueName: \"kubernetes.io/projected/1d474c1c-0534-4149-b026-e32b23029845-kube-api-access-d6x7l\") pod \"cert-manager-cainjector-559cdd8c87-4g56j\" (UID: \"1d474c1c-0534-4149-b026-e32b23029845\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4g56j" Jul 20 17:58:20.192075 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:20.192051 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d474c1c-0534-4149-b026-e32b23029845-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4g56j\" (UID: \"1d474c1c-0534-4149-b026-e32b23029845\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4g56j" Jul 20 17:58:20.307608 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:20.307574 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4g56j" Jul 20 17:58:20.367865 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:20.367835 2583 generic.go:356] "Generic (PLEG): container finished" podID="d6ee24b4-538a-4e89-abfb-b99f7dde3b9c" containerID="46aef9c66aba80b333e9d1e8a89fc445ecef9e2766a44df8532aa4ade1551f6c" exitCode=0 Jul 20 17:58:20.368025 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:20.367950 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs" event={"ID":"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c","Type":"ContainerDied","Data":"46aef9c66aba80b333e9d1e8a89fc445ecef9e2766a44df8532aa4ade1551f6c"} Jul 20 17:58:20.371201 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:20.371170 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-ptjpn" event={"ID":"2ce2898c-dda5-44f2-8a60-0f742e434807","Type":"ContainerStarted","Data":"3f09add9cffd241c017fbe33eb0cd9dbdb868543b35109c0d4ef6e34ae91d801"} Jul 20 17:58:20.372592 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:20.372568 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-ptjpn" Jul 20 17:58:20.429440 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:20.429395 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-ptjpn" podStartSLOduration=1.589469241 podStartE2EDuration="4.429379786s" totalImagesPullingTime="2.839910545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:58:20.427926657 +0000 UTC m=+608.799715230" watchObservedRunningTime="2026-07-20 17:58:20.429379786 +0000 UTC m=+608.801168360" Jul 20 17:58:20.455244 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:20.455221 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4g56j"] Jul 20 17:58:20.456924 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:58:20.456897 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d474c1c_0534_4149_b026_e32b23029845.slice/crio-34c6145849736ac242ef46f1fb6f0c1789534e61759c537e8dcd62968e58e84e WatchSource:0}: Error finding container 34c6145849736ac242ef46f1fb6f0c1789534e61759c537e8dcd62968e58e84e: Status 404 returned error can't find the container with id 34c6145849736ac242ef46f1fb6f0c1789534e61759c537e8dcd62968e58e84e Jul 20 17:58:21.377693 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:21.377663 2583 generic.go:356] "Generic (PLEG): container finished" podID="d6ee24b4-538a-4e89-abfb-b99f7dde3b9c" containerID="79df004aae9afda9088ad32b9b3ec3120dafe628d22b6b2d5b8b87b2ce72ca31" exitCode=0 Jul 20 17:58:21.378119 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:21.377755 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs" event={"ID":"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c","Type":"ContainerDied","Data":"79df004aae9afda9088ad32b9b3ec3120dafe628d22b6b2d5b8b87b2ce72ca31"} Jul 20 17:58:21.379250 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:21.379229 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4g56j" event={"ID":"1d474c1c-0534-4149-b026-e32b23029845","Type":"ContainerStarted","Data":"93c702f217aae5e258f37ae97a147368aef4ad1598f1809686ff8cd2cae8486a"} Jul 20 17:58:21.379349 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:21.379263 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4g56j" event={"ID":"1d474c1c-0534-4149-b026-e32b23029845","Type":"ContainerStarted","Data":"34c6145849736ac242ef46f1fb6f0c1789534e61759c537e8dcd62968e58e84e"} Jul 20 17:58:21.441582 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:21.441531 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4g56j" podStartSLOduration=2.441515263 podStartE2EDuration="2.441515263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:58:21.437761705 +0000 UTC m=+609.809550279" watchObservedRunningTime="2026-07-20 17:58:21.441515263 +0000 UTC m=+609.813303836" Jul 20 17:58:22.509477 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:22.509454 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs" Jul 20 17:58:22.602226 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:22.602190 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6ee24b4-538a-4e89-abfb-b99f7dde3b9c-bundle\") pod \"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c\" (UID: \"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c\") " Jul 20 17:58:22.602420 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:22.602241 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6ee24b4-538a-4e89-abfb-b99f7dde3b9c-util\") pod \"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c\" (UID: \"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c\") " Jul 20 17:58:22.602420 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:22.602301 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khwm4\" (UniqueName: \"kubernetes.io/projected/d6ee24b4-538a-4e89-abfb-b99f7dde3b9c-kube-api-access-khwm4\") pod \"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c\" (UID: \"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c\") " Jul 20 17:58:22.602683 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:22.602661 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ee24b4-538a-4e89-abfb-b99f7dde3b9c-bundle" (OuterVolumeSpecName: "bundle") pod "d6ee24b4-538a-4e89-abfb-b99f7dde3b9c" (UID: "d6ee24b4-538a-4e89-abfb-b99f7dde3b9c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 17:58:22.604567 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:22.604540 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ee24b4-538a-4e89-abfb-b99f7dde3b9c-kube-api-access-khwm4" (OuterVolumeSpecName: "kube-api-access-khwm4") pod "d6ee24b4-538a-4e89-abfb-b99f7dde3b9c" (UID: "d6ee24b4-538a-4e89-abfb-b99f7dde3b9c"). InnerVolumeSpecName "kube-api-access-khwm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 17:58:22.606412 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:22.606389 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ee24b4-538a-4e89-abfb-b99f7dde3b9c-util" (OuterVolumeSpecName: "util") pod "d6ee24b4-538a-4e89-abfb-b99f7dde3b9c" (UID: "d6ee24b4-538a-4e89-abfb-b99f7dde3b9c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 17:58:22.703209 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:22.703176 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khwm4\" (UniqueName: \"kubernetes.io/projected/d6ee24b4-538a-4e89-abfb-b99f7dde3b9c-kube-api-access-khwm4\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 17:58:22.703209 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:22.703203 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6ee24b4-538a-4e89-abfb-b99f7dde3b9c-bundle\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 17:58:22.703209 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:22.703213 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6ee24b4-538a-4e89-abfb-b99f7dde3b9c-util\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 17:58:23.389330 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:23.389291 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs" event={"ID":"d6ee24b4-538a-4e89-abfb-b99f7dde3b9c","Type":"ContainerDied","Data":"ad6653f40db4cf4428d7f5b49cd4f8cc280d70597a1c2159c26c9e07a6ff8986"} Jul 20 17:58:23.389330 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:23.389311 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2m5xs" Jul 20 17:58:23.389528 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:23.389341 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad6653f40db4cf4428d7f5b49cd4f8cc280d70597a1c2159c26c9e07a6ff8986" Jul 20 17:58:26.381194 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:26.381164 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-ptjpn" Jul 20 17:58:28.153287 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.153257 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6mxjw"] Jul 20 17:58:28.153660 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.153536 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ee24b4-538a-4e89-abfb-b99f7dde3b9c" containerName="extract" Jul 20 17:58:28.153660 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.153547 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ee24b4-538a-4e89-abfb-b99f7dde3b9c" containerName="extract" Jul 20 17:58:28.153660 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.153561 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ee24b4-538a-4e89-abfb-b99f7dde3b9c" containerName="util" Jul 20 17:58:28.153660 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.153568 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ee24b4-538a-4e89-abfb-b99f7dde3b9c" containerName="util" Jul 20 17:58:28.153660 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.153577 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ee24b4-538a-4e89-abfb-b99f7dde3b9c" containerName="pull" Jul 20 17:58:28.153660 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.153585 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ee24b4-538a-4e89-abfb-b99f7dde3b9c" containerName="pull" Jul 20 17:58:28.153660 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.153644 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6ee24b4-538a-4e89-abfb-b99f7dde3b9c" containerName="extract" Jul 20 17:58:28.157049 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.157034 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6mxjw" Jul 20 17:58:28.159598 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.159578 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-7bwhg\"" Jul 20 17:58:28.170135 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.170115 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6mxjw"] Jul 20 17:58:28.251620 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.251591 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26a0d330-bc05-4abc-9fd9-c028baa52c10-bound-sa-token\") pod \"cert-manager-55b4d954d5-6mxjw\" (UID: \"26a0d330-bc05-4abc-9fd9-c028baa52c10\") " pod="cert-manager/cert-manager-55b4d954d5-6mxjw" Jul 20 17:58:28.251772 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.251638 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pczvf\" (UniqueName: \"kubernetes.io/projected/26a0d330-bc05-4abc-9fd9-c028baa52c10-kube-api-access-pczvf\") pod \"cert-manager-55b4d954d5-6mxjw\" (UID: \"26a0d330-bc05-4abc-9fd9-c028baa52c10\") " pod="cert-manager/cert-manager-55b4d954d5-6mxjw" Jul 20 17:58:28.352448 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.352413 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pczvf\" (UniqueName: \"kubernetes.io/projected/26a0d330-bc05-4abc-9fd9-c028baa52c10-kube-api-access-pczvf\") pod \"cert-manager-55b4d954d5-6mxjw\" (UID: \"26a0d330-bc05-4abc-9fd9-c028baa52c10\") " pod="cert-manager/cert-manager-55b4d954d5-6mxjw" Jul 20 17:58:28.352591 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.352474 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26a0d330-bc05-4abc-9fd9-c028baa52c10-bound-sa-token\") pod \"cert-manager-55b4d954d5-6mxjw\" (UID: \"26a0d330-bc05-4abc-9fd9-c028baa52c10\") " pod="cert-manager/cert-manager-55b4d954d5-6mxjw" Jul 20 17:58:28.369005 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.368976 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pczvf\" (UniqueName: \"kubernetes.io/projected/26a0d330-bc05-4abc-9fd9-c028baa52c10-kube-api-access-pczvf\") pod \"cert-manager-55b4d954d5-6mxjw\" (UID: \"26a0d330-bc05-4abc-9fd9-c028baa52c10\") " pod="cert-manager/cert-manager-55b4d954d5-6mxjw" Jul 20 17:58:28.369150 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.369127 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26a0d330-bc05-4abc-9fd9-c028baa52c10-bound-sa-token\") pod \"cert-manager-55b4d954d5-6mxjw\" (UID: \"26a0d330-bc05-4abc-9fd9-c028baa52c10\") " pod="cert-manager/cert-manager-55b4d954d5-6mxjw" Jul 20 17:58:28.465672 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.465646 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6mxjw" Jul 20 17:58:28.592672 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:28.592645 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6mxjw"] Jul 20 17:58:28.594629 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:58:28.594596 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26a0d330_bc05_4abc_9fd9_c028baa52c10.slice/crio-1193af922de8c1a37da3f4d41cca78f551520f930b9353ab7395dad14d39b3e3 WatchSource:0}: Error finding container 1193af922de8c1a37da3f4d41cca78f551520f930b9353ab7395dad14d39b3e3: Status 404 returned error can't find the container with id 1193af922de8c1a37da3f4d41cca78f551520f930b9353ab7395dad14d39b3e3 Jul 20 17:58:29.414764 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:29.414702 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6mxjw" event={"ID":"26a0d330-bc05-4abc-9fd9-c028baa52c10","Type":"ContainerStarted","Data":"2f3f64afb22004fcf82194da1297cfe15651c7266ffd79753977a071bba66f1f"} Jul 20 17:58:29.414764 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:29.414759 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6mxjw" event={"ID":"26a0d330-bc05-4abc-9fd9-c028baa52c10","Type":"ContainerStarted","Data":"1193af922de8c1a37da3f4d41cca78f551520f930b9353ab7395dad14d39b3e3"} Jul 20 17:58:29.433636 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:29.433591 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-6mxjw" podStartSLOduration=1.433575561 podStartE2EDuration="1.433575561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:58:29.433378674 +0000 UTC m=+617.805167247" watchObservedRunningTime="2026-07-20 17:58:29.433575561 +0000 UTC m=+617.805364134" Jul 20 17:58:40.461485 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:40.461407 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh"] Jul 20 17:58:40.463776 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:40.463760 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh" Jul 20 17:58:40.467711 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:40.467691 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 20 17:58:40.467903 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:40.467878 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-t695n\"" Jul 20 17:58:40.468426 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:40.468411 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 20 17:58:40.478396 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:40.478369 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh"] Jul 20 17:58:40.653400 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:40.653369 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sf99\" (UniqueName: \"kubernetes.io/projected/5566cff1-5be2-42a6-a05c-80217ba5ecfb-kube-api-access-9sf99\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh\" (UID: \"5566cff1-5be2-42a6-a05c-80217ba5ecfb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh" Jul 20 17:58:40.653400 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:40.653401 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5566cff1-5be2-42a6-a05c-80217ba5ecfb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh\" (UID: \"5566cff1-5be2-42a6-a05c-80217ba5ecfb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh" Jul 20 17:58:40.653609 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:40.653470 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5566cff1-5be2-42a6-a05c-80217ba5ecfb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh\" (UID: \"5566cff1-5be2-42a6-a05c-80217ba5ecfb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh" Jul 20 17:58:40.754876 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:40.754783 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5566cff1-5be2-42a6-a05c-80217ba5ecfb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh\" (UID: \"5566cff1-5be2-42a6-a05c-80217ba5ecfb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh" Jul 20 17:58:40.754876 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:40.754843 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sf99\" (UniqueName: \"kubernetes.io/projected/5566cff1-5be2-42a6-a05c-80217ba5ecfb-kube-api-access-9sf99\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh\" (UID: \"5566cff1-5be2-42a6-a05c-80217ba5ecfb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh" Jul 20 17:58:40.754876 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:40.754867 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5566cff1-5be2-42a6-a05c-80217ba5ecfb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh\" (UID: \"5566cff1-5be2-42a6-a05c-80217ba5ecfb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh" Jul 20 17:58:40.755158 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:40.755136 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5566cff1-5be2-42a6-a05c-80217ba5ecfb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh\" (UID: \"5566cff1-5be2-42a6-a05c-80217ba5ecfb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh" Jul 20 17:58:40.755232 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:40.755210 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5566cff1-5be2-42a6-a05c-80217ba5ecfb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh\" (UID: \"5566cff1-5be2-42a6-a05c-80217ba5ecfb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh" Jul 20 17:58:40.764760 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:40.764718 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sf99\" (UniqueName: \"kubernetes.io/projected/5566cff1-5be2-42a6-a05c-80217ba5ecfb-kube-api-access-9sf99\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh\" (UID: \"5566cff1-5be2-42a6-a05c-80217ba5ecfb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh" Jul 20 17:58:40.773369 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:40.773351 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh" Jul 20 17:58:40.909200 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:40.909177 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh"] Jul 20 17:58:40.911103 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:58:40.911081 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5566cff1_5be2_42a6_a05c_80217ba5ecfb.slice/crio-df44115a1a43d89dca100ee5c98786e583399b4adaf439c3902ddc6194d061de WatchSource:0}: Error finding container df44115a1a43d89dca100ee5c98786e583399b4adaf439c3902ddc6194d061de: Status 404 returned error can't find the container with id df44115a1a43d89dca100ee5c98786e583399b4adaf439c3902ddc6194d061de Jul 20 17:58:41.465953 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:41.465922 2583 generic.go:356] "Generic (PLEG): container finished" podID="5566cff1-5be2-42a6-a05c-80217ba5ecfb" containerID="17661459b04cfefe019e904db4e3d3ecead46068828ef182583fe76bcd7812a5" exitCode=0 Jul 20 17:58:41.466351 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:41.466005 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh" event={"ID":"5566cff1-5be2-42a6-a05c-80217ba5ecfb","Type":"ContainerDied","Data":"17661459b04cfefe019e904db4e3d3ecead46068828ef182583fe76bcd7812a5"} Jul 20 17:58:41.466351 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:41.466057 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh" event={"ID":"5566cff1-5be2-42a6-a05c-80217ba5ecfb","Type":"ContainerStarted","Data":"df44115a1a43d89dca100ee5c98786e583399b4adaf439c3902ddc6194d061de"} Jul 20 17:58:42.472013 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:42.471982 2583 generic.go:356] "Generic (PLEG): container finished" podID="5566cff1-5be2-42a6-a05c-80217ba5ecfb" containerID="d71f0d8a476d5ff24939c37295feaacc2116e7e75d9ff968afd1f862d1d96f1c" exitCode=0 Jul 20 17:58:42.472402 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:42.472071 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh" event={"ID":"5566cff1-5be2-42a6-a05c-80217ba5ecfb","Type":"ContainerDied","Data":"d71f0d8a476d5ff24939c37295feaacc2116e7e75d9ff968afd1f862d1d96f1c"} Jul 20 17:58:43.478935 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:43.478904 2583 generic.go:356] "Generic (PLEG): container finished" podID="5566cff1-5be2-42a6-a05c-80217ba5ecfb" containerID="f4b8a81b3e8f862d88e75e35ce729ed20316343a1e6499a2d54d1b0718dc0a8e" exitCode=0 Jul 20 17:58:43.479282 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:43.478956 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh" event={"ID":"5566cff1-5be2-42a6-a05c-80217ba5ecfb","Type":"ContainerDied","Data":"f4b8a81b3e8f862d88e75e35ce729ed20316343a1e6499a2d54d1b0718dc0a8e"} Jul 20 17:58:44.605129 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:44.605107 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh" Jul 20 17:58:44.682922 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:44.682889 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sf99\" (UniqueName: \"kubernetes.io/projected/5566cff1-5be2-42a6-a05c-80217ba5ecfb-kube-api-access-9sf99\") pod \"5566cff1-5be2-42a6-a05c-80217ba5ecfb\" (UID: \"5566cff1-5be2-42a6-a05c-80217ba5ecfb\") " Jul 20 17:58:44.683110 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:44.682933 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5566cff1-5be2-42a6-a05c-80217ba5ecfb-util\") pod \"5566cff1-5be2-42a6-a05c-80217ba5ecfb\" (UID: \"5566cff1-5be2-42a6-a05c-80217ba5ecfb\") " Jul 20 17:58:44.683110 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:44.682966 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5566cff1-5be2-42a6-a05c-80217ba5ecfb-bundle\") pod \"5566cff1-5be2-42a6-a05c-80217ba5ecfb\" (UID: \"5566cff1-5be2-42a6-a05c-80217ba5ecfb\") " Jul 20 17:58:44.683901 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:44.683873 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5566cff1-5be2-42a6-a05c-80217ba5ecfb-bundle" (OuterVolumeSpecName: "bundle") pod "5566cff1-5be2-42a6-a05c-80217ba5ecfb" (UID: "5566cff1-5be2-42a6-a05c-80217ba5ecfb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 17:58:44.685042 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:44.685012 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5566cff1-5be2-42a6-a05c-80217ba5ecfb-kube-api-access-9sf99" (OuterVolumeSpecName: "kube-api-access-9sf99") pod "5566cff1-5be2-42a6-a05c-80217ba5ecfb" (UID: "5566cff1-5be2-42a6-a05c-80217ba5ecfb"). InnerVolumeSpecName "kube-api-access-9sf99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 17:58:44.688441 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:44.688410 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5566cff1-5be2-42a6-a05c-80217ba5ecfb-util" (OuterVolumeSpecName: "util") pod "5566cff1-5be2-42a6-a05c-80217ba5ecfb" (UID: "5566cff1-5be2-42a6-a05c-80217ba5ecfb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 17:58:44.784000 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:44.783930 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sf99\" (UniqueName: \"kubernetes.io/projected/5566cff1-5be2-42a6-a05c-80217ba5ecfb-kube-api-access-9sf99\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 17:58:44.784000 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:44.783954 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5566cff1-5be2-42a6-a05c-80217ba5ecfb-util\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 17:58:44.784000 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:44.783963 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5566cff1-5be2-42a6-a05c-80217ba5ecfb-bundle\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 17:58:45.491112 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:45.491084 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh" Jul 20 17:58:45.491112 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:45.491109 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqt6wh" event={"ID":"5566cff1-5be2-42a6-a05c-80217ba5ecfb","Type":"ContainerDied","Data":"df44115a1a43d89dca100ee5c98786e583399b4adaf439c3902ddc6194d061de"} Jul 20 17:58:45.491343 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:45.491149 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df44115a1a43d89dca100ee5c98786e583399b4adaf439c3902ddc6194d061de" Jul 20 17:58:52.998974 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:52.998939 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb"] Jul 20 17:58:52.999573 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:52.999369 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5566cff1-5be2-42a6-a05c-80217ba5ecfb" containerName="pull" Jul 20 17:58:52.999573 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:52.999388 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5566cff1-5be2-42a6-a05c-80217ba5ecfb" containerName="pull" Jul 20 17:58:52.999573 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:52.999399 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5566cff1-5be2-42a6-a05c-80217ba5ecfb" containerName="extract" Jul 20 17:58:52.999573 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:52.999408 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5566cff1-5be2-42a6-a05c-80217ba5ecfb" containerName="extract" Jul 20 17:58:52.999573 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:52.999444 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5566cff1-5be2-42a6-a05c-80217ba5ecfb" containerName="util" Jul 20 17:58:52.999573 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:52.999452 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5566cff1-5be2-42a6-a05c-80217ba5ecfb" containerName="util" Jul 20 17:58:52.999573 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:52.999524 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5566cff1-5be2-42a6-a05c-80217ba5ecfb" containerName="extract" Jul 20 17:58:53.001842 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.001818 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:58:53.004802 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.004781 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 20 17:58:53.008925 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.008910 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 20 17:58:53.009024 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.008925 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 20 17:58:53.009191 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.009179 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 20 17:58:53.012849 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.012834 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jtcl9\"" Jul 20 17:58:53.034363 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.034336 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb"] Jul 20 17:58:53.044048 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.044028 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8eda8dbf-bc27-40e6-8f4a-698616904858-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-c4qlb\" (UID: \"8eda8dbf-bc27-40e6-8f4a-698616904858\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:58:53.044165 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.044059 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8eda8dbf-bc27-40e6-8f4a-698616904858-manifests\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-c4qlb\" (UID: \"8eda8dbf-bc27-40e6-8f4a-698616904858\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:58:53.044165 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.044089 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmcj6\" (UniqueName: \"kubernetes.io/projected/8eda8dbf-bc27-40e6-8f4a-698616904858-kube-api-access-cmcj6\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-c4qlb\" (UID: \"8eda8dbf-bc27-40e6-8f4a-698616904858\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:58:53.044275 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.044191 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8eda8dbf-bc27-40e6-8f4a-698616904858-tmp\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-c4qlb\" (UID: \"8eda8dbf-bc27-40e6-8f4a-698616904858\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:58:53.044275 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.044226 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8eda8dbf-bc27-40e6-8f4a-698616904858-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-c4qlb\" (UID: \"8eda8dbf-bc27-40e6-8f4a-698616904858\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:58:53.144969 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.144935 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8eda8dbf-bc27-40e6-8f4a-698616904858-tmp\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-c4qlb\" (UID: \"8eda8dbf-bc27-40e6-8f4a-698616904858\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:58:53.144969 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.144971 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8eda8dbf-bc27-40e6-8f4a-698616904858-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-c4qlb\" (UID: \"8eda8dbf-bc27-40e6-8f4a-698616904858\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:58:53.145156 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.145102 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8eda8dbf-bc27-40e6-8f4a-698616904858-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-c4qlb\" (UID: \"8eda8dbf-bc27-40e6-8f4a-698616904858\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:58:53.145156 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.145142 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8eda8dbf-bc27-40e6-8f4a-698616904858-manifests\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-c4qlb\" (UID: \"8eda8dbf-bc27-40e6-8f4a-698616904858\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:58:53.145233 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.145190 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmcj6\" (UniqueName: \"kubernetes.io/projected/8eda8dbf-bc27-40e6-8f4a-698616904858-kube-api-access-cmcj6\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-c4qlb\" (UID: \"8eda8dbf-bc27-40e6-8f4a-698616904858\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:58:53.145568 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.145540 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8eda8dbf-bc27-40e6-8f4a-698616904858-manifests\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-c4qlb\" (UID: \"8eda8dbf-bc27-40e6-8f4a-698616904858\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:58:53.147252 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.147231 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8eda8dbf-bc27-40e6-8f4a-698616904858-tmp\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-c4qlb\" (UID: \"8eda8dbf-bc27-40e6-8f4a-698616904858\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:58:53.147538 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.147520 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8eda8dbf-bc27-40e6-8f4a-698616904858-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-c4qlb\" (UID: \"8eda8dbf-bc27-40e6-8f4a-698616904858\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:58:53.147618 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.147588 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8eda8dbf-bc27-40e6-8f4a-698616904858-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-c4qlb\" (UID: \"8eda8dbf-bc27-40e6-8f4a-698616904858\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:58:53.166720 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.166696 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmcj6\" (UniqueName: \"kubernetes.io/projected/8eda8dbf-bc27-40e6-8f4a-698616904858-kube-api-access-cmcj6\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-c4qlb\" (UID: \"8eda8dbf-bc27-40e6-8f4a-698616904858\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:58:53.311624 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.311545 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:58:53.458921 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:58:53.458889 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eda8dbf_bc27_40e6_8f4a_698616904858.slice/crio-5775ce3e3fb4754f0fc38cc3c74c003e8d864076734c74167c25acf3cf5cf9fa WatchSource:0}: Error finding container 5775ce3e3fb4754f0fc38cc3c74c003e8d864076734c74167c25acf3cf5cf9fa: Status 404 returned error can't find the container with id 5775ce3e3fb4754f0fc38cc3c74c003e8d864076734c74167c25acf3cf5cf9fa Jul 20 17:58:53.459097 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.459081 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb"] Jul 20 17:58:53.532047 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:53.532010 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" event={"ID":"8eda8dbf-bc27-40e6-8f4a-698616904858","Type":"ContainerStarted","Data":"5775ce3e3fb4754f0fc38cc3c74c003e8d864076734c74167c25acf3cf5cf9fa"} Jul 20 17:58:56.547844 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:56.547803 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" event={"ID":"8eda8dbf-bc27-40e6-8f4a-698616904858","Type":"ContainerStarted","Data":"fa936c943b44bacd6b23ba9b1f3b03c5141c658949895403ca8073f813395c4b"} Jul 20 17:58:57.553633 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:57.553600 2583 generic.go:356] "Generic (PLEG): container finished" podID="8eda8dbf-bc27-40e6-8f4a-698616904858" containerID="fa936c943b44bacd6b23ba9b1f3b03c5141c658949895403ca8073f813395c4b" exitCode=0 Jul 20 17:58:57.554013 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:57.553684 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" event={"ID":"8eda8dbf-bc27-40e6-8f4a-698616904858","Type":"ContainerDied","Data":"fa936c943b44bacd6b23ba9b1f3b03c5141c658949895403ca8073f813395c4b"} Jul 20 17:58:58.560544 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:58.560505 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" event={"ID":"8eda8dbf-bc27-40e6-8f4a-698616904858","Type":"ContainerStarted","Data":"7de7fd3651529f8865646a1d70298fc8173242a5f9c776116c70d9a32a75601e"} Jul 20 17:58:58.560986 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:58.560654 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:58:58.589156 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:58:58.589111 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" podStartSLOduration=2.294407533 podStartE2EDuration="6.589097337s" totalImagesPullingTime="3.206876048s" totalInitContainerRuntime="1.087813756s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:58:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:58:58.586081411 +0000 UTC m=+646.957869985" watchObservedRunningTime="2026-07-20 17:58:58.589097337 +0000 UTC m=+646.960885911" Jul 20 17:59:09.566817 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:09.566780 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-c4qlb" Jul 20 17:59:11.512830 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.512787 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7974845875-rvrrz"] Jul 20 17:59:11.581597 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.581563 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7974845875-rvrrz"] Jul 20 17:59:11.581772 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.581675 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" Jul 20 17:59:11.587583 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.587563 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 20 17:59:11.588939 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.588911 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 20 17:59:11.590121 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.590103 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 20 17:59:11.590223 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.590110 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 20 17:59:11.590383 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.590368 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2kkf2\"" Jul 20 17:59:11.590476 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.590372 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 20 17:59:11.696680 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.696643 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfb742af-fb9a-4d7f-b199-2d0f3d54fa11-cert\") pod \"lws-controller-manager-7974845875-rvrrz\" (UID: \"cfb742af-fb9a-4d7f-b199-2d0f3d54fa11\") " pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" Jul 20 17:59:11.696883 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.696697 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cfb742af-fb9a-4d7f-b199-2d0f3d54fa11-manager-config\") pod \"lws-controller-manager-7974845875-rvrrz\" (UID: \"cfb742af-fb9a-4d7f-b199-2d0f3d54fa11\") " pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" Jul 20 17:59:11.696883 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.696745 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfb742af-fb9a-4d7f-b199-2d0f3d54fa11-metrics-cert\") pod \"lws-controller-manager-7974845875-rvrrz\" (UID: \"cfb742af-fb9a-4d7f-b199-2d0f3d54fa11\") " pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" Jul 20 17:59:11.696883 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.696805 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4gfj\" (UniqueName: \"kubernetes.io/projected/cfb742af-fb9a-4d7f-b199-2d0f3d54fa11-kube-api-access-c4gfj\") pod \"lws-controller-manager-7974845875-rvrrz\" (UID: \"cfb742af-fb9a-4d7f-b199-2d0f3d54fa11\") " pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" Jul 20 17:59:11.797576 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.797506 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfb742af-fb9a-4d7f-b199-2d0f3d54fa11-cert\") pod \"lws-controller-manager-7974845875-rvrrz\" (UID: \"cfb742af-fb9a-4d7f-b199-2d0f3d54fa11\") " pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" Jul 20 17:59:11.797576 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.797550 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cfb742af-fb9a-4d7f-b199-2d0f3d54fa11-manager-config\") pod \"lws-controller-manager-7974845875-rvrrz\" (UID: \"cfb742af-fb9a-4d7f-b199-2d0f3d54fa11\") " pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" Jul 20 17:59:11.797827 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.797583 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfb742af-fb9a-4d7f-b199-2d0f3d54fa11-metrics-cert\") pod \"lws-controller-manager-7974845875-rvrrz\" (UID: \"cfb742af-fb9a-4d7f-b199-2d0f3d54fa11\") " pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" Jul 20 17:59:11.797827 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.797623 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4gfj\" (UniqueName: \"kubernetes.io/projected/cfb742af-fb9a-4d7f-b199-2d0f3d54fa11-kube-api-access-c4gfj\") pod \"lws-controller-manager-7974845875-rvrrz\" (UID: \"cfb742af-fb9a-4d7f-b199-2d0f3d54fa11\") " pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" Jul 20 17:59:11.798247 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.798224 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cfb742af-fb9a-4d7f-b199-2d0f3d54fa11-manager-config\") pod \"lws-controller-manager-7974845875-rvrrz\" (UID: \"cfb742af-fb9a-4d7f-b199-2d0f3d54fa11\") " pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" Jul 20 17:59:11.800091 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.800066 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfb742af-fb9a-4d7f-b199-2d0f3d54fa11-cert\") pod \"lws-controller-manager-7974845875-rvrrz\" (UID: \"cfb742af-fb9a-4d7f-b199-2d0f3d54fa11\") " pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" Jul 20 17:59:11.800185 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.800102 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfb742af-fb9a-4d7f-b199-2d0f3d54fa11-metrics-cert\") pod \"lws-controller-manager-7974845875-rvrrz\" (UID: \"cfb742af-fb9a-4d7f-b199-2d0f3d54fa11\") " pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" Jul 20 17:59:11.834195 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.834171 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4gfj\" (UniqueName: \"kubernetes.io/projected/cfb742af-fb9a-4d7f-b199-2d0f3d54fa11-kube-api-access-c4gfj\") pod \"lws-controller-manager-7974845875-rvrrz\" (UID: \"cfb742af-fb9a-4d7f-b199-2d0f3d54fa11\") " pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" Jul 20 17:59:11.891089 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:11.891050 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" Jul 20 17:59:12.098211 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:12.098187 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7974845875-rvrrz"] Jul 20 17:59:12.099963 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:59:12.099938 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb742af_fb9a_4d7f_b199_2d0f3d54fa11.slice/crio-126d38cfccc7fd79ad109cfeb8ae05488e2e7206d166c5a9e0c232475fe04ceb WatchSource:0}: Error finding container 126d38cfccc7fd79ad109cfeb8ae05488e2e7206d166c5a9e0c232475fe04ceb: Status 404 returned error can't find the container with id 126d38cfccc7fd79ad109cfeb8ae05488e2e7206d166c5a9e0c232475fe04ceb Jul 20 17:59:12.624779 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:12.624739 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" event={"ID":"cfb742af-fb9a-4d7f-b199-2d0f3d54fa11","Type":"ContainerStarted","Data":"126d38cfccc7fd79ad109cfeb8ae05488e2e7206d166c5a9e0c232475fe04ceb"} Jul 20 17:59:13.640215 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:13.640167 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" event={"ID":"cfb742af-fb9a-4d7f-b199-2d0f3d54fa11","Type":"ContainerStarted","Data":"62c524d7160c610435e6b36ca75b6d7052eec799fb5cbdcb6e9eb34f71ea1017"} Jul 20 17:59:13.640704 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:13.640304 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" Jul 20 17:59:13.666220 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:13.666164 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" podStartSLOduration=1.329736299 podStartE2EDuration="2.666147478s" totalImagesPullingTime="1.336411179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:59:13.664498601 +0000 UTC m=+662.036287201" watchObservedRunningTime="2026-07-20 17:59:13.666147478 +0000 UTC m=+662.037936051" Jul 20 17:59:21.635857 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.633537 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw"] Jul 20 17:59:21.639317 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.639283 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:21.641924 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:59:21.641892 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"istiod-tls\" is forbidden: User \"system:node:ip-10-0-141-129.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-129.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" type="*v1.Secret" Jul 20 17:59:21.642438 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:59:21.642414 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"istiod-openshift-gateway-94bb8fbfd-z4njw\" is forbidden: User \"system:node:ip-10-0-141-129.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-129.ec2.internal' and this object" podUID="01389904-5991-4a81-9f66-7148e67defd0" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:21.642965 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:59:21.642941 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-gw-ca-root-cert\" is forbidden: User \"system:node:ip-10-0-141-129.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-129.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" Jul 20 17:59:21.643059 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:59:21.642956 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"istio-kubeconfig\" is forbidden: User \"system:node:ip-10-0-141-129.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-129.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret" Jul 20 17:59:21.643212 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:59:21.643195 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cacerts\" is forbidden: User \"system:node:ip-10-0-141-129.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-129.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"cacerts\"" type="*v1.Secret" Jul 20 17:59:21.643273 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:59:21.643225 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"istiod-openshift-gateway-dockercfg-888w8\" is forbidden: User \"system:node:ip-10-0-141-129.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-129.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-888w8\"" type="*v1.Secret" Jul 20 17:59:21.643492 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:59:21.643469 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-129.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-129.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 20 17:59:21.643707 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:59:21.643691 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-129.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-129.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 20 17:59:21.657097 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.657072 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw"] Jul 20 17:59:21.781389 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.781351 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/01389904-5991-4a81-9f66-7148e67defd0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:21.781572 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.781404 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01389904-5991-4a81-9f66-7148e67defd0-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:21.781572 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.781421 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/01389904-5991-4a81-9f66-7148e67defd0-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:21.781572 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.781532 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01389904-5991-4a81-9f66-7148e67defd0-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:21.781572 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.781560 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps64g\" (UniqueName: \"kubernetes.io/projected/01389904-5991-4a81-9f66-7148e67defd0-kube-api-access-ps64g\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:21.781719 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.781587 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/01389904-5991-4a81-9f66-7148e67defd0-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:21.781719 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.781634 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/01389904-5991-4a81-9f66-7148e67defd0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:21.882215 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.882176 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01389904-5991-4a81-9f66-7148e67defd0-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:21.882215 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.882215 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps64g\" (UniqueName: \"kubernetes.io/projected/01389904-5991-4a81-9f66-7148e67defd0-kube-api-access-ps64g\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:21.882475 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.882234 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/01389904-5991-4a81-9f66-7148e67defd0-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:21.882475 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.882257 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/01389904-5991-4a81-9f66-7148e67defd0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:21.882475 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.882290 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/01389904-5991-4a81-9f66-7148e67defd0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:21.882475 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.882340 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01389904-5991-4a81-9f66-7148e67defd0-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:21.882475 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.882364 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/01389904-5991-4a81-9f66-7148e67defd0-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:21.884486 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.884462 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/01389904-5991-4a81-9f66-7148e67defd0-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:21.891001 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:21.890939 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01389904-5991-4a81-9f66-7148e67defd0-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:22.495529 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:22.495493 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 20 17:59:22.503404 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:22.503381 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/01389904-5991-4a81-9f66-7148e67defd0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:22.546653 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:22.546631 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 20 17:59:22.623143 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:22.623116 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 20 17:59:22.632356 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:22.632332 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps64g\" (UniqueName: \"kubernetes.io/projected/01389904-5991-4a81-9f66-7148e67defd0-kube-api-access-ps64g\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:22.805893 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:22.805816 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 20 17:59:22.815371 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:22.815350 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/01389904-5991-4a81-9f66-7148e67defd0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:22.882529 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:59:22.882502 2583 secret.go:189] Couldn't get secret openshift-ingress/istio-kubeconfig: failed to sync secret cache: timed out waiting for the condition Jul 20 17:59:22.882689 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:59:22.882567 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01389904-5991-4a81-9f66-7148e67defd0-istio-kubeconfig podName:01389904-5991-4a81-9f66-7148e67defd0 nodeName:}" failed. No retries permitted until 2026-07-20 17:59:23.382551188 +0000 UTC m=+671.754339740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "istio-kubeconfig" (UniqueName: "kubernetes.io/secret/01389904-5991-4a81-9f66-7148e67defd0-istio-kubeconfig") pod "istiod-openshift-gateway-94bb8fbfd-z4njw" (UID: "01389904-5991-4a81-9f66-7148e67defd0") : failed to sync secret cache: timed out waiting for the condition Jul 20 17:59:22.882689 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:59:22.882498 2583 secret.go:189] Couldn't get secret openshift-ingress/cacerts: failed to sync secret cache: timed out waiting for the condition Jul 20 17:59:22.882689 ip-10-0-141-129 kubenswrapper[2583]: E0720 17:59:22.882600 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01389904-5991-4a81-9f66-7148e67defd0-cacerts podName:01389904-5991-4a81-9f66-7148e67defd0 nodeName:}" failed. No retries permitted until 2026-07-20 17:59:23.382590812 +0000 UTC m=+671.754379363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cacerts" (UniqueName: "kubernetes.io/secret/01389904-5991-4a81-9f66-7148e67defd0-cacerts") pod "istiod-openshift-gateway-94bb8fbfd-z4njw" (UID: "01389904-5991-4a81-9f66-7148e67defd0") : failed to sync secret cache: timed out waiting for the condition Jul 20 17:59:23.030690 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:23.030661 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 20 17:59:23.063038 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:23.062971 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-888w8\"" Jul 20 17:59:23.066760 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:23.066720 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 20 17:59:23.396260 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:23.396178 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/01389904-5991-4a81-9f66-7148e67defd0-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:23.396260 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:23.396233 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01389904-5991-4a81-9f66-7148e67defd0-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:23.398813 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:23.398754 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01389904-5991-4a81-9f66-7148e67defd0-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:23.398958 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:23.398928 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/01389904-5991-4a81-9f66-7148e67defd0-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-z4njw\" (UID: \"01389904-5991-4a81-9f66-7148e67defd0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:23.449013 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:23.448985 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:23.645591 ip-10-0-141-129 kubenswrapper[2583]: W0720 17:59:23.645562 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01389904_5991_4a81_9f66_7148e67defd0.slice/crio-5fad0c159b6c66b641f66a3c7a63794af555c0fb59d35eba33522d6281a343af WatchSource:0}: Error finding container 5fad0c159b6c66b641f66a3c7a63794af555c0fb59d35eba33522d6281a343af: Status 404 returned error can't find the container with id 5fad0c159b6c66b641f66a3c7a63794af555c0fb59d35eba33522d6281a343af Jul 20 17:59:23.659562 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:23.659531 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw"] Jul 20 17:59:23.686106 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:23.686068 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" event={"ID":"01389904-5991-4a81-9f66-7148e67defd0","Type":"ContainerStarted","Data":"5fad0c159b6c66b641f66a3c7a63794af555c0fb59d35eba33522d6281a343af"} Jul 20 17:59:24.647597 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:24.647558 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" Jul 20 17:59:25.965194 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:25.965158 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 17:59:25.965431 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:25.965224 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 17:59:26.704104 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:26.704073 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" event={"ID":"01389904-5991-4a81-9f66-7148e67defd0","Type":"ContainerStarted","Data":"c6df4c7038e8645ed4cab902eca53f4e79980747d4a8129fc12236b6319f9095"} Jul 20 17:59:26.704305 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:26.704195 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 17:59:26.747920 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:26.747850 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" podStartSLOduration=3.430423029 podStartE2EDuration="5.74782812s" totalImagesPullingTime="2.317405091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:59:26.744720845 +0000 UTC m=+675.116509423" watchObservedRunningTime="2026-07-20 17:59:26.74782812 +0000 UTC m=+675.119616695" Jul 20 17:59:27.710636 ip-10-0-141-129 kubenswrapper[2583]: I0720 17:59:27.710608 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z4njw" Jul 20 18:00:29.678567 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:29.678485 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f"] Jul 20 18:00:29.681121 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:29.681105 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f" Jul 20 18:00:29.683439 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:29.683419 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 20 18:00:29.683574 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:29.683482 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-559lz\"" Jul 20 18:00:29.684190 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:29.684175 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 20 18:00:29.689826 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:29.689806 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f"] Jul 20 18:00:29.833370 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:29.833333 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbe65489-22b0-4198-b374-ed587a6a6d01-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f\" (UID: \"dbe65489-22b0-4198-b374-ed587a6a6d01\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f" Jul 20 18:00:29.833554 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:29.833378 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf7sv\" (UniqueName: \"kubernetes.io/projected/dbe65489-22b0-4198-b374-ed587a6a6d01-kube-api-access-jf7sv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f\" (UID: \"dbe65489-22b0-4198-b374-ed587a6a6d01\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f" Jul 20 18:00:29.833554 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:29.833423 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbe65489-22b0-4198-b374-ed587a6a6d01-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f\" (UID: \"dbe65489-22b0-4198-b374-ed587a6a6d01\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f" Jul 20 18:00:29.933965 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:29.933938 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbe65489-22b0-4198-b374-ed587a6a6d01-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f\" (UID: \"dbe65489-22b0-4198-b374-ed587a6a6d01\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f" Jul 20 18:00:29.934105 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:29.933979 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf7sv\" (UniqueName: \"kubernetes.io/projected/dbe65489-22b0-4198-b374-ed587a6a6d01-kube-api-access-jf7sv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f\" (UID: \"dbe65489-22b0-4198-b374-ed587a6a6d01\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f" Jul 20 18:00:29.934105 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:29.934016 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbe65489-22b0-4198-b374-ed587a6a6d01-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f\" (UID: \"dbe65489-22b0-4198-b374-ed587a6a6d01\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f" Jul 20 18:00:29.934318 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:29.934298 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbe65489-22b0-4198-b374-ed587a6a6d01-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f\" (UID: \"dbe65489-22b0-4198-b374-ed587a6a6d01\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f" Jul 20 18:00:29.934360 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:29.934347 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbe65489-22b0-4198-b374-ed587a6a6d01-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f\" (UID: \"dbe65489-22b0-4198-b374-ed587a6a6d01\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f" Jul 20 18:00:29.942708 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:29.942682 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf7sv\" (UniqueName: \"kubernetes.io/projected/dbe65489-22b0-4198-b374-ed587a6a6d01-kube-api-access-jf7sv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f\" (UID: \"dbe65489-22b0-4198-b374-ed587a6a6d01\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f" Jul 20 18:00:29.991178 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:29.991154 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f" Jul 20 18:00:30.121485 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.121462 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f"] Jul 20 18:00:30.123476 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:00:30.123448 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe65489_22b0_4198_b374_ed587a6a6d01.slice/crio-3beb5ea57c96fce1a2a8962e2f049444145629ad812a1c95a8b595af98bfc7a6 WatchSource:0}: Error finding container 3beb5ea57c96fce1a2a8962e2f049444145629ad812a1c95a8b595af98bfc7a6: Status 404 returned error can't find the container with id 3beb5ea57c96fce1a2a8962e2f049444145629ad812a1c95a8b595af98bfc7a6 Jul 20 18:00:30.277945 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.277908 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5"] Jul 20 18:00:30.280276 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.280253 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5" Jul 20 18:00:30.289153 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.289121 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5"] Jul 20 18:00:30.438645 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.438613 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51dfed50-bc82-4dc1-ad65-ab84ced8504a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5\" (UID: \"51dfed50-bc82-4dc1-ad65-ab84ced8504a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5" Jul 20 18:00:30.438852 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.438676 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51dfed50-bc82-4dc1-ad65-ab84ced8504a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5\" (UID: \"51dfed50-bc82-4dc1-ad65-ab84ced8504a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5" Jul 20 18:00:30.438852 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.438706 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb48w\" (UniqueName: \"kubernetes.io/projected/51dfed50-bc82-4dc1-ad65-ab84ced8504a-kube-api-access-qb48w\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5\" (UID: \"51dfed50-bc82-4dc1-ad65-ab84ced8504a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5" Jul 20 18:00:30.539190 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.539106 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51dfed50-bc82-4dc1-ad65-ab84ced8504a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5\" (UID: \"51dfed50-bc82-4dc1-ad65-ab84ced8504a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5" Jul 20 18:00:30.539190 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.539173 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51dfed50-bc82-4dc1-ad65-ab84ced8504a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5\" (UID: \"51dfed50-bc82-4dc1-ad65-ab84ced8504a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5" Jul 20 18:00:30.539364 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.539204 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb48w\" (UniqueName: \"kubernetes.io/projected/51dfed50-bc82-4dc1-ad65-ab84ced8504a-kube-api-access-qb48w\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5\" (UID: \"51dfed50-bc82-4dc1-ad65-ab84ced8504a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5" Jul 20 18:00:30.539504 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.539485 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51dfed50-bc82-4dc1-ad65-ab84ced8504a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5\" (UID: \"51dfed50-bc82-4dc1-ad65-ab84ced8504a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5" Jul 20 18:00:30.539592 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.539575 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51dfed50-bc82-4dc1-ad65-ab84ced8504a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5\" (UID: \"51dfed50-bc82-4dc1-ad65-ab84ced8504a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5" Jul 20 18:00:30.547118 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.547092 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb48w\" (UniqueName: \"kubernetes.io/projected/51dfed50-bc82-4dc1-ad65-ab84ced8504a-kube-api-access-qb48w\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5\" (UID: \"51dfed50-bc82-4dc1-ad65-ab84ced8504a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5" Jul 20 18:00:30.612678 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.612654 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5" Jul 20 18:00:30.741515 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.741488 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5"] Jul 20 18:00:30.743770 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:00:30.743722 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51dfed50_bc82_4dc1_ad65_ab84ced8504a.slice/crio-4c434def4cbedf02fda6230042bd2749c0b805732fe81d6f666388fcef7394a0 WatchSource:0}: Error finding container 4c434def4cbedf02fda6230042bd2749c0b805732fe81d6f666388fcef7394a0: Status 404 returned error can't find the container with id 4c434def4cbedf02fda6230042bd2749c0b805732fe81d6f666388fcef7394a0 Jul 20 18:00:30.878941 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.878906 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr"] Jul 20 18:00:30.881174 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.881159 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr" Jul 20 18:00:30.890438 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.890409 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr"] Jul 20 18:00:30.993011 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.992977 2583 generic.go:356] "Generic (PLEG): container finished" podID="dbe65489-22b0-4198-b374-ed587a6a6d01" containerID="f5ad8232a85dae34d6e0d81f6584a8d2217502f5889cc6ea39c4c901f02072ed" exitCode=0 Jul 20 18:00:30.993171 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.993063 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f" event={"ID":"dbe65489-22b0-4198-b374-ed587a6a6d01","Type":"ContainerDied","Data":"f5ad8232a85dae34d6e0d81f6584a8d2217502f5889cc6ea39c4c901f02072ed"} Jul 20 18:00:30.993171 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.993108 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f" event={"ID":"dbe65489-22b0-4198-b374-ed587a6a6d01","Type":"ContainerStarted","Data":"3beb5ea57c96fce1a2a8962e2f049444145629ad812a1c95a8b595af98bfc7a6"} Jul 20 18:00:30.994643 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.994593 2583 generic.go:356] "Generic (PLEG): container finished" podID="51dfed50-bc82-4dc1-ad65-ab84ced8504a" containerID="6e232317d4a44b04a64ea748a605f3613ec6baec7884b5a0dafd86b36dc2865d" exitCode=0 Jul 20 18:00:30.994748 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.994660 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5" event={"ID":"51dfed50-bc82-4dc1-ad65-ab84ced8504a","Type":"ContainerDied","Data":"6e232317d4a44b04a64ea748a605f3613ec6baec7884b5a0dafd86b36dc2865d"} Jul 20 18:00:30.994748 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:30.994687 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5" event={"ID":"51dfed50-bc82-4dc1-ad65-ab84ced8504a","Type":"ContainerStarted","Data":"4c434def4cbedf02fda6230042bd2749c0b805732fe81d6f666388fcef7394a0"} Jul 20 18:00:31.043386 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.043358 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzgp4\" (UniqueName: \"kubernetes.io/projected/84002a57-b378-4606-8064-155b0b23bff1-kube-api-access-tzgp4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr\" (UID: \"84002a57-b378-4606-8064-155b0b23bff1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr" Jul 20 18:00:31.043386 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.043388 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84002a57-b378-4606-8064-155b0b23bff1-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr\" (UID: \"84002a57-b378-4606-8064-155b0b23bff1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr" Jul 20 18:00:31.043554 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.043420 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84002a57-b378-4606-8064-155b0b23bff1-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr\" (UID: \"84002a57-b378-4606-8064-155b0b23bff1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr" Jul 20 18:00:31.144714 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.144641 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzgp4\" (UniqueName: \"kubernetes.io/projected/84002a57-b378-4606-8064-155b0b23bff1-kube-api-access-tzgp4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr\" (UID: \"84002a57-b378-4606-8064-155b0b23bff1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr" Jul 20 18:00:31.144714 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.144683 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84002a57-b378-4606-8064-155b0b23bff1-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr\" (UID: \"84002a57-b378-4606-8064-155b0b23bff1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr" Jul 20 18:00:31.144893 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.144828 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84002a57-b378-4606-8064-155b0b23bff1-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr\" (UID: \"84002a57-b378-4606-8064-155b0b23bff1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr" Jul 20 18:00:31.145241 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.145220 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84002a57-b378-4606-8064-155b0b23bff1-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr\" (UID: \"84002a57-b378-4606-8064-155b0b23bff1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr" Jul 20 18:00:31.145241 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.145232 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84002a57-b378-4606-8064-155b0b23bff1-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr\" (UID: \"84002a57-b378-4606-8064-155b0b23bff1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr" Jul 20 18:00:31.153139 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.153109 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzgp4\" (UniqueName: \"kubernetes.io/projected/84002a57-b378-4606-8064-155b0b23bff1-kube-api-access-tzgp4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr\" (UID: \"84002a57-b378-4606-8064-155b0b23bff1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr" Jul 20 18:00:31.203828 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.203796 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr" Jul 20 18:00:31.332484 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.332462 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr"] Jul 20 18:00:31.334670 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:00:31.334646 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84002a57_b378_4606_8064_155b0b23bff1.slice/crio-c5346c02d58d128168ba4334996de18926d55e1f5e2cc07e71f75e73f93421bc WatchSource:0}: Error finding container c5346c02d58d128168ba4334996de18926d55e1f5e2cc07e71f75e73f93421bc: Status 404 returned error can't find the container with id c5346c02d58d128168ba4334996de18926d55e1f5e2cc07e71f75e73f93421bc Jul 20 18:00:31.484841 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.484811 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn"] Jul 20 18:00:31.487335 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.487317 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn" Jul 20 18:00:31.499091 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.498788 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn"] Jul 20 18:00:31.649614 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.649575 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b183506-1a15-4994-905f-88d0ee42c349-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn\" (UID: \"0b183506-1a15-4994-905f-88d0ee42c349\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn" Jul 20 18:00:31.649810 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.649624 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b183506-1a15-4994-905f-88d0ee42c349-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn\" (UID: \"0b183506-1a15-4994-905f-88d0ee42c349\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn" Jul 20 18:00:31.649810 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.649700 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnzkh\" (UniqueName: \"kubernetes.io/projected/0b183506-1a15-4994-905f-88d0ee42c349-kube-api-access-dnzkh\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn\" (UID: \"0b183506-1a15-4994-905f-88d0ee42c349\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn" Jul 20 18:00:31.751142 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.751026 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnzkh\" (UniqueName: \"kubernetes.io/projected/0b183506-1a15-4994-905f-88d0ee42c349-kube-api-access-dnzkh\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn\" (UID: \"0b183506-1a15-4994-905f-88d0ee42c349\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn" Jul 20 18:00:31.751142 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.751114 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b183506-1a15-4994-905f-88d0ee42c349-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn\" (UID: \"0b183506-1a15-4994-905f-88d0ee42c349\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn" Jul 20 18:00:31.751142 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.751141 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b183506-1a15-4994-905f-88d0ee42c349-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn\" (UID: \"0b183506-1a15-4994-905f-88d0ee42c349\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn" Jul 20 18:00:31.751614 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.751500 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b183506-1a15-4994-905f-88d0ee42c349-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn\" (UID: \"0b183506-1a15-4994-905f-88d0ee42c349\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn" Jul 20 18:00:31.751614 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.751527 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b183506-1a15-4994-905f-88d0ee42c349-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn\" (UID: \"0b183506-1a15-4994-905f-88d0ee42c349\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn" Jul 20 18:00:31.759513 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.759493 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnzkh\" (UniqueName: \"kubernetes.io/projected/0b183506-1a15-4994-905f-88d0ee42c349-kube-api-access-dnzkh\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn\" (UID: \"0b183506-1a15-4994-905f-88d0ee42c349\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn" Jul 20 18:00:31.801029 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.800994 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn" Jul 20 18:00:31.933686 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:31.933657 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn"] Jul 20 18:00:31.935297 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:00:31.935270 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b183506_1a15_4994_905f_88d0ee42c349.slice/crio-821a6737fa287d1559b412ea864426d8e8d7f2b505828ad8d53ec371c186ac07 WatchSource:0}: Error finding container 821a6737fa287d1559b412ea864426d8e8d7f2b505828ad8d53ec371c186ac07: Status 404 returned error can't find the container with id 821a6737fa287d1559b412ea864426d8e8d7f2b505828ad8d53ec371c186ac07 Jul 20 18:00:32.002098 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:32.002038 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn" event={"ID":"0b183506-1a15-4994-905f-88d0ee42c349","Type":"ContainerStarted","Data":"821a6737fa287d1559b412ea864426d8e8d7f2b505828ad8d53ec371c186ac07"} Jul 20 18:00:32.003626 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:32.003603 2583 generic.go:356] "Generic (PLEG): container finished" podID="84002a57-b378-4606-8064-155b0b23bff1" containerID="c211a0ebb75f9c711919dfc229d8ad0b2d0c4e8bff88289223bea2e7a82a8db8" exitCode=0 Jul 20 18:00:32.003754 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:32.003639 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr" event={"ID":"84002a57-b378-4606-8064-155b0b23bff1","Type":"ContainerDied","Data":"c211a0ebb75f9c711919dfc229d8ad0b2d0c4e8bff88289223bea2e7a82a8db8"} Jul 20 18:00:32.003754 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:32.003678 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr" event={"ID":"84002a57-b378-4606-8064-155b0b23bff1","Type":"ContainerStarted","Data":"c5346c02d58d128168ba4334996de18926d55e1f5e2cc07e71f75e73f93421bc"} Jul 20 18:00:33.010188 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:33.010104 2583 generic.go:356] "Generic (PLEG): container finished" podID="0b183506-1a15-4994-905f-88d0ee42c349" containerID="569d48dfc707536c13f68e616f2ec181d808d185577935a523e7b7d7c96b21b2" exitCode=0 Jul 20 18:00:33.010589 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:33.010192 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn" event={"ID":"0b183506-1a15-4994-905f-88d0ee42c349","Type":"ContainerDied","Data":"569d48dfc707536c13f68e616f2ec181d808d185577935a523e7b7d7c96b21b2"} Jul 20 18:00:33.011974 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:33.011955 2583 generic.go:356] "Generic (PLEG): container finished" podID="51dfed50-bc82-4dc1-ad65-ab84ced8504a" containerID="3a83d35853e6770886fb08ab80840ba6fe22ead32ed452b28bdcaf35734c8ca0" exitCode=0 Jul 20 18:00:33.012079 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:33.012023 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5" event={"ID":"51dfed50-bc82-4dc1-ad65-ab84ced8504a","Type":"ContainerDied","Data":"3a83d35853e6770886fb08ab80840ba6fe22ead32ed452b28bdcaf35734c8ca0"} Jul 20 18:00:33.013846 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:33.013824 2583 generic.go:356] "Generic (PLEG): container finished" podID="84002a57-b378-4606-8064-155b0b23bff1" containerID="d5ad74ace5df2758214b60c6fd05956f0daa6fa5b1d6c654ef8d43fcde3f12d3" exitCode=0 Jul 20 18:00:33.013963 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:33.013910 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr" event={"ID":"84002a57-b378-4606-8064-155b0b23bff1","Type":"ContainerDied","Data":"d5ad74ace5df2758214b60c6fd05956f0daa6fa5b1d6c654ef8d43fcde3f12d3"} Jul 20 18:00:34.020561 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:34.020526 2583 generic.go:356] "Generic (PLEG): container finished" podID="51dfed50-bc82-4dc1-ad65-ab84ced8504a" containerID="d26a3ded9529d55d8fbf6535a3948b4000259803edf8f5610383578fe808cf1a" exitCode=0 Jul 20 18:00:34.020890 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:34.020602 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5" event={"ID":"51dfed50-bc82-4dc1-ad65-ab84ced8504a","Type":"ContainerDied","Data":"d26a3ded9529d55d8fbf6535a3948b4000259803edf8f5610383578fe808cf1a"} Jul 20 18:00:34.022515 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:34.022495 2583 generic.go:356] "Generic (PLEG): container finished" podID="84002a57-b378-4606-8064-155b0b23bff1" containerID="9491b1d094ad2780386aac2dea0ff6c1ee5e51b0a97d2f7211e6b5da5122600c" exitCode=0 Jul 20 18:00:34.022600 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:34.022574 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr" event={"ID":"84002a57-b378-4606-8064-155b0b23bff1","Type":"ContainerDied","Data":"9491b1d094ad2780386aac2dea0ff6c1ee5e51b0a97d2f7211e6b5da5122600c"} Jul 20 18:00:35.029142 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.029111 2583 generic.go:356] "Generic (PLEG): container finished" podID="0b183506-1a15-4994-905f-88d0ee42c349" containerID="b48fbb89140ba39836c6d06a4c5e5b2d6d39309c25c3474d3b1d54b27429eae6" exitCode=0 Jul 20 18:00:35.029594 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.029203 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn" event={"ID":"0b183506-1a15-4994-905f-88d0ee42c349","Type":"ContainerDied","Data":"b48fbb89140ba39836c6d06a4c5e5b2d6d39309c25c3474d3b1d54b27429eae6"} Jul 20 18:00:35.031103 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.031083 2583 generic.go:356] "Generic (PLEG): container finished" podID="dbe65489-22b0-4198-b374-ed587a6a6d01" containerID="3f03e752c7827b51b173f9d89c75cf9d4c7ba46f819243388dcc22ec9827848f" exitCode=0 Jul 20 18:00:35.031187 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.031160 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f" event={"ID":"dbe65489-22b0-4198-b374-ed587a6a6d01","Type":"ContainerDied","Data":"3f03e752c7827b51b173f9d89c75cf9d4c7ba46f819243388dcc22ec9827848f"} Jul 20 18:00:35.181950 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.181863 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr" Jul 20 18:00:35.203685 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.203663 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5" Jul 20 18:00:35.283177 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.283119 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84002a57-b378-4606-8064-155b0b23bff1-util\") pod \"84002a57-b378-4606-8064-155b0b23bff1\" (UID: \"84002a57-b378-4606-8064-155b0b23bff1\") " Jul 20 18:00:35.283177 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.283149 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb48w\" (UniqueName: \"kubernetes.io/projected/51dfed50-bc82-4dc1-ad65-ab84ced8504a-kube-api-access-qb48w\") pod \"51dfed50-bc82-4dc1-ad65-ab84ced8504a\" (UID: \"51dfed50-bc82-4dc1-ad65-ab84ced8504a\") " Jul 20 18:00:35.283376 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.283192 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84002a57-b378-4606-8064-155b0b23bff1-bundle\") pod \"84002a57-b378-4606-8064-155b0b23bff1\" (UID: \"84002a57-b378-4606-8064-155b0b23bff1\") " Jul 20 18:00:35.283376 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.283217 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzgp4\" (UniqueName: \"kubernetes.io/projected/84002a57-b378-4606-8064-155b0b23bff1-kube-api-access-tzgp4\") pod \"84002a57-b378-4606-8064-155b0b23bff1\" (UID: \"84002a57-b378-4606-8064-155b0b23bff1\") " Jul 20 18:00:35.283376 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.283255 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51dfed50-bc82-4dc1-ad65-ab84ced8504a-bundle\") pod \"51dfed50-bc82-4dc1-ad65-ab84ced8504a\" (UID: \"51dfed50-bc82-4dc1-ad65-ab84ced8504a\") " Jul 20 18:00:35.283376 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.283290 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51dfed50-bc82-4dc1-ad65-ab84ced8504a-util\") pod \"51dfed50-bc82-4dc1-ad65-ab84ced8504a\" (UID: \"51dfed50-bc82-4dc1-ad65-ab84ced8504a\") " Jul 20 18:00:35.283787 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.283723 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84002a57-b378-4606-8064-155b0b23bff1-bundle" (OuterVolumeSpecName: "bundle") pod "84002a57-b378-4606-8064-155b0b23bff1" (UID: "84002a57-b378-4606-8064-155b0b23bff1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:00:35.283905 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.283790 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51dfed50-bc82-4dc1-ad65-ab84ced8504a-bundle" (OuterVolumeSpecName: "bundle") pod "51dfed50-bc82-4dc1-ad65-ab84ced8504a" (UID: "51dfed50-bc82-4dc1-ad65-ab84ced8504a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:00:35.285485 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.285461 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51dfed50-bc82-4dc1-ad65-ab84ced8504a-kube-api-access-qb48w" (OuterVolumeSpecName: "kube-api-access-qb48w") pod "51dfed50-bc82-4dc1-ad65-ab84ced8504a" (UID: "51dfed50-bc82-4dc1-ad65-ab84ced8504a"). InnerVolumeSpecName "kube-api-access-qb48w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:00:35.285673 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.285650 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84002a57-b378-4606-8064-155b0b23bff1-kube-api-access-tzgp4" (OuterVolumeSpecName: "kube-api-access-tzgp4") pod "84002a57-b378-4606-8064-155b0b23bff1" (UID: "84002a57-b378-4606-8064-155b0b23bff1"). InnerVolumeSpecName "kube-api-access-tzgp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:00:35.289344 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.289319 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51dfed50-bc82-4dc1-ad65-ab84ced8504a-util" (OuterVolumeSpecName: "util") pod "51dfed50-bc82-4dc1-ad65-ab84ced8504a" (UID: "51dfed50-bc82-4dc1-ad65-ab84ced8504a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:00:35.289522 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.289502 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84002a57-b378-4606-8064-155b0b23bff1-util" (OuterVolumeSpecName: "util") pod "84002a57-b378-4606-8064-155b0b23bff1" (UID: "84002a57-b378-4606-8064-155b0b23bff1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:00:35.384072 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.384034 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84002a57-b378-4606-8064-155b0b23bff1-util\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:00:35.384072 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.384068 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb48w\" (UniqueName: \"kubernetes.io/projected/51dfed50-bc82-4dc1-ad65-ab84ced8504a-kube-api-access-qb48w\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:00:35.384072 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.384079 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84002a57-b378-4606-8064-155b0b23bff1-bundle\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:00:35.384072 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.384088 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzgp4\" (UniqueName: \"kubernetes.io/projected/84002a57-b378-4606-8064-155b0b23bff1-kube-api-access-tzgp4\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:00:35.384300 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.384095 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51dfed50-bc82-4dc1-ad65-ab84ced8504a-bundle\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:00:35.384300 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:35.384103 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51dfed50-bc82-4dc1-ad65-ab84ced8504a-util\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:00:36.037440 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:36.037412 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5" Jul 20 18:00:36.037892 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:36.037407 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5" event={"ID":"51dfed50-bc82-4dc1-ad65-ab84ced8504a","Type":"ContainerDied","Data":"4c434def4cbedf02fda6230042bd2749c0b805732fe81d6f666388fcef7394a0"} Jul 20 18:00:36.037892 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:36.037522 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c434def4cbedf02fda6230042bd2749c0b805732fe81d6f666388fcef7394a0" Jul 20 18:00:36.039160 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:36.039142 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr" Jul 20 18:00:36.039294 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:36.039175 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr" event={"ID":"84002a57-b378-4606-8064-155b0b23bff1","Type":"ContainerDied","Data":"c5346c02d58d128168ba4334996de18926d55e1f5e2cc07e71f75e73f93421bc"} Jul 20 18:00:36.039294 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:36.039214 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5346c02d58d128168ba4334996de18926d55e1f5e2cc07e71f75e73f93421bc" Jul 20 18:00:36.041378 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:36.041358 2583 generic.go:356] "Generic (PLEG): container finished" podID="0b183506-1a15-4994-905f-88d0ee42c349" containerID="7a5f7ca2afe0f1986a0d69a48b7254212fd08940bac97ab2703989a488b2b991" exitCode=0 Jul 20 18:00:36.041486 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:36.041434 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn" event={"ID":"0b183506-1a15-4994-905f-88d0ee42c349","Type":"ContainerDied","Data":"7a5f7ca2afe0f1986a0d69a48b7254212fd08940bac97ab2703989a488b2b991"} Jul 20 18:00:36.043593 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:36.043571 2583 generic.go:356] "Generic (PLEG): container finished" podID="dbe65489-22b0-4198-b374-ed587a6a6d01" containerID="c086a1a7c4ac4a408b6ab13bf18dd61fe9266e88889b79a7def78fbd68957ece" exitCode=0 Jul 20 18:00:36.043672 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:36.043619 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f" event={"ID":"dbe65489-22b0-4198-b374-ed587a6a6d01","Type":"ContainerDied","Data":"c086a1a7c4ac4a408b6ab13bf18dd61fe9266e88889b79a7def78fbd68957ece"} Jul 20 18:00:37.180714 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.180689 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f" Jul 20 18:00:37.214854 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.214831 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn" Jul 20 18:00:37.300870 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.300802 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b183506-1a15-4994-905f-88d0ee42c349-util\") pod \"0b183506-1a15-4994-905f-88d0ee42c349\" (UID: \"0b183506-1a15-4994-905f-88d0ee42c349\") " Jul 20 18:00:37.300870 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.300854 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbe65489-22b0-4198-b374-ed587a6a6d01-util\") pod \"dbe65489-22b0-4198-b374-ed587a6a6d01\" (UID: \"dbe65489-22b0-4198-b374-ed587a6a6d01\") " Jul 20 18:00:37.301080 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.300883 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b183506-1a15-4994-905f-88d0ee42c349-bundle\") pod \"0b183506-1a15-4994-905f-88d0ee42c349\" (UID: \"0b183506-1a15-4994-905f-88d0ee42c349\") " Jul 20 18:00:37.301080 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.300912 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jf7sv\" (UniqueName: \"kubernetes.io/projected/dbe65489-22b0-4198-b374-ed587a6a6d01-kube-api-access-jf7sv\") pod \"dbe65489-22b0-4198-b374-ed587a6a6d01\" (UID: \"dbe65489-22b0-4198-b374-ed587a6a6d01\") " Jul 20 18:00:37.301080 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.300943 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnzkh\" (UniqueName: \"kubernetes.io/projected/0b183506-1a15-4994-905f-88d0ee42c349-kube-api-access-dnzkh\") pod \"0b183506-1a15-4994-905f-88d0ee42c349\" (UID: \"0b183506-1a15-4994-905f-88d0ee42c349\") " Jul 20 18:00:37.301080 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.300967 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbe65489-22b0-4198-b374-ed587a6a6d01-bundle\") pod \"dbe65489-22b0-4198-b374-ed587a6a6d01\" (UID: \"dbe65489-22b0-4198-b374-ed587a6a6d01\") " Jul 20 18:00:37.301585 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.301532 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe65489-22b0-4198-b374-ed587a6a6d01-bundle" (OuterVolumeSpecName: "bundle") pod "dbe65489-22b0-4198-b374-ed587a6a6d01" (UID: "dbe65489-22b0-4198-b374-ed587a6a6d01"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:00:37.301681 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.301643 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b183506-1a15-4994-905f-88d0ee42c349-bundle" (OuterVolumeSpecName: "bundle") pod "0b183506-1a15-4994-905f-88d0ee42c349" (UID: "0b183506-1a15-4994-905f-88d0ee42c349"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:00:37.303519 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.303493 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b183506-1a15-4994-905f-88d0ee42c349-kube-api-access-dnzkh" (OuterVolumeSpecName: "kube-api-access-dnzkh") pod "0b183506-1a15-4994-905f-88d0ee42c349" (UID: "0b183506-1a15-4994-905f-88d0ee42c349"). InnerVolumeSpecName "kube-api-access-dnzkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:00:37.303519 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.303510 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe65489-22b0-4198-b374-ed587a6a6d01-kube-api-access-jf7sv" (OuterVolumeSpecName: "kube-api-access-jf7sv") pod "dbe65489-22b0-4198-b374-ed587a6a6d01" (UID: "dbe65489-22b0-4198-b374-ed587a6a6d01"). InnerVolumeSpecName "kube-api-access-jf7sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:00:37.305848 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.305828 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b183506-1a15-4994-905f-88d0ee42c349-util" (OuterVolumeSpecName: "util") pod "0b183506-1a15-4994-905f-88d0ee42c349" (UID: "0b183506-1a15-4994-905f-88d0ee42c349"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:00:37.305966 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.305948 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe65489-22b0-4198-b374-ed587a6a6d01-util" (OuterVolumeSpecName: "util") pod "dbe65489-22b0-4198-b374-ed587a6a6d01" (UID: "dbe65489-22b0-4198-b374-ed587a6a6d01"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:00:37.402034 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.402005 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b183506-1a15-4994-905f-88d0ee42c349-util\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:00:37.402034 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.402028 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbe65489-22b0-4198-b374-ed587a6a6d01-util\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:00:37.402034 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.402036 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b183506-1a15-4994-905f-88d0ee42c349-bundle\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:00:37.402214 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.402045 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf7sv\" (UniqueName: \"kubernetes.io/projected/dbe65489-22b0-4198-b374-ed587a6a6d01-kube-api-access-jf7sv\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:00:37.402214 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.402055 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnzkh\" (UniqueName: \"kubernetes.io/projected/0b183506-1a15-4994-905f-88d0ee42c349-kube-api-access-dnzkh\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:00:37.402214 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:37.402063 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbe65489-22b0-4198-b374-ed587a6a6d01-bundle\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:00:38.054753 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:38.054704 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn" Jul 20 18:00:38.054753 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:38.054708 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn" event={"ID":"0b183506-1a15-4994-905f-88d0ee42c349","Type":"ContainerDied","Data":"821a6737fa287d1559b412ea864426d8e8d7f2b505828ad8d53ec371c186ac07"} Jul 20 18:00:38.054998 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:38.054775 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="821a6737fa287d1559b412ea864426d8e8d7f2b505828ad8d53ec371c186ac07" Jul 20 18:00:38.056517 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:38.056501 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f" Jul 20 18:00:38.056616 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:38.056523 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f" event={"ID":"dbe65489-22b0-4198-b374-ed587a6a6d01","Type":"ContainerDied","Data":"3beb5ea57c96fce1a2a8962e2f049444145629ad812a1c95a8b595af98bfc7a6"} Jul 20 18:00:38.056616 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:00:38.056551 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3beb5ea57c96fce1a2a8962e2f049444145629ad812a1c95a8b595af98bfc7a6" Jul 20 18:02:03.748660 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.748627 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.748931 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51dfed50-bc82-4dc1-ad65-ab84ced8504a" containerName="util" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.748942 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="51dfed50-bc82-4dc1-ad65-ab84ced8504a" containerName="util" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.748953 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe65489-22b0-4198-b374-ed587a6a6d01" containerName="util" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.748959 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe65489-22b0-4198-b374-ed587a6a6d01" containerName="util" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.748968 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b183506-1a15-4994-905f-88d0ee42c349" containerName="pull" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.748975 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b183506-1a15-4994-905f-88d0ee42c349" containerName="pull" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.748984 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51dfed50-bc82-4dc1-ad65-ab84ced8504a" containerName="pull" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.748989 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="51dfed50-bc82-4dc1-ad65-ab84ced8504a" containerName="pull" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.748996 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84002a57-b378-4606-8064-155b0b23bff1" containerName="util" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749001 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="84002a57-b378-4606-8064-155b0b23bff1" containerName="util" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749008 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84002a57-b378-4606-8064-155b0b23bff1" containerName="pull" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749013 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="84002a57-b378-4606-8064-155b0b23bff1" containerName="pull" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749019 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe65489-22b0-4198-b374-ed587a6a6d01" containerName="extract" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749023 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe65489-22b0-4198-b374-ed587a6a6d01" containerName="extract" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749030 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b183506-1a15-4994-905f-88d0ee42c349" containerName="extract" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749035 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b183506-1a15-4994-905f-88d0ee42c349" containerName="extract" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749043 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b183506-1a15-4994-905f-88d0ee42c349" containerName="util" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749050 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b183506-1a15-4994-905f-88d0ee42c349" containerName="util" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749057 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51dfed50-bc82-4dc1-ad65-ab84ced8504a" containerName="extract" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749063 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="51dfed50-bc82-4dc1-ad65-ab84ced8504a" containerName="extract" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749069 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe65489-22b0-4198-b374-ed587a6a6d01" containerName="pull" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749075 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe65489-22b0-4198-b374-ed587a6a6d01" containerName="pull" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749084 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84002a57-b378-4606-8064-155b0b23bff1" containerName="extract" Jul 20 18:02:03.749113 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749089 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="84002a57-b378-4606-8064-155b0b23bff1" containerName="extract" Jul 20 18:02:03.749873 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749134 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b183506-1a15-4994-905f-88d0ee42c349" containerName="extract" Jul 20 18:02:03.749873 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749143 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="51dfed50-bc82-4dc1-ad65-ab84ced8504a" containerName="extract" Jul 20 18:02:03.749873 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749150 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbe65489-22b0-4198-b374-ed587a6a6d01" containerName="extract" Jul 20 18:02:03.749873 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.749155 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="84002a57-b378-4606-8064-155b0b23bff1" containerName="extract" Jul 20 18:02:03.751872 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.751838 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-h76fd" Jul 20 18:02:03.754412 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.754386 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 20 18:02:03.755291 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.755273 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-559lz\"" Jul 20 18:02:03.755377 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.755275 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 20 18:02:03.755377 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.755314 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 20 18:02:03.764249 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.764230 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:02:03.782444 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.782424 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e37cc553-2ea6-485e-ba1e-f4ca1001dd5e-config-file\") pod \"limitador-limitador-54d6d988d5-h76fd\" (UID: \"e37cc553-2ea6-485e-ba1e-f4ca1001dd5e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-h76fd" Jul 20 18:02:03.782548 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.782452 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtgtz\" (UniqueName: \"kubernetes.io/projected/e37cc553-2ea6-485e-ba1e-f4ca1001dd5e-kube-api-access-rtgtz\") pod \"limitador-limitador-54d6d988d5-h76fd\" (UID: \"e37cc553-2ea6-485e-ba1e-f4ca1001dd5e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-h76fd" Jul 20 18:02:03.787167 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.787147 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:02:03.882980 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.882935 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e37cc553-2ea6-485e-ba1e-f4ca1001dd5e-config-file\") pod \"limitador-limitador-54d6d988d5-h76fd\" (UID: \"e37cc553-2ea6-485e-ba1e-f4ca1001dd5e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-h76fd" Jul 20 18:02:03.882980 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.882990 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtgtz\" (UniqueName: \"kubernetes.io/projected/e37cc553-2ea6-485e-ba1e-f4ca1001dd5e-kube-api-access-rtgtz\") pod \"limitador-limitador-54d6d988d5-h76fd\" (UID: \"e37cc553-2ea6-485e-ba1e-f4ca1001dd5e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-h76fd" Jul 20 18:02:03.883615 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.883596 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e37cc553-2ea6-485e-ba1e-f4ca1001dd5e-config-file\") pod \"limitador-limitador-54d6d988d5-h76fd\" (UID: \"e37cc553-2ea6-485e-ba1e-f4ca1001dd5e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-h76fd" Jul 20 18:02:03.898175 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:03.898141 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtgtz\" (UniqueName: \"kubernetes.io/projected/e37cc553-2ea6-485e-ba1e-f4ca1001dd5e-kube-api-access-rtgtz\") pod \"limitador-limitador-54d6d988d5-h76fd\" (UID: \"e37cc553-2ea6-485e-ba1e-f4ca1001dd5e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-h76fd" Jul 20 18:02:04.062521 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:04.062439 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-h76fd" Jul 20 18:02:04.212944 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:04.212913 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:02:04.214561 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:02:04.214536 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37cc553_2ea6_485e_ba1e_f4ca1001dd5e.slice/crio-770ef20835b103bde40aa01be658eb8373a35fd2888edd2f940720c2c35eb233 WatchSource:0}: Error finding container 770ef20835b103bde40aa01be658eb8373a35fd2888edd2f940720c2c35eb233: Status 404 returned error can't find the container with id 770ef20835b103bde40aa01be658eb8373a35fd2888edd2f940720c2c35eb233 Jul 20 18:02:04.457177 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:04.457141 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-h76fd" event={"ID":"e37cc553-2ea6-485e-ba1e-f4ca1001dd5e","Type":"ContainerStarted","Data":"770ef20835b103bde40aa01be658eb8373a35fd2888edd2f940720c2c35eb233"} Jul 20 18:02:07.474600 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:07.474559 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-h76fd" event={"ID":"e37cc553-2ea6-485e-ba1e-f4ca1001dd5e","Type":"ContainerStarted","Data":"148d0ee61334a7d6892aef47c84cee1b8ebf9de3d8be717faf5df91c25ffd40c"} Jul 20 18:02:07.475080 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:07.474618 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-h76fd" Jul 20 18:02:07.493650 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:07.493610 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-h76fd" podStartSLOduration=2.051559701 podStartE2EDuration="4.493595912s" totalImagesPullingTime="2.442036211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:02:07.492099541 +0000 UTC m=+835.863888115" watchObservedRunningTime="2026-07-20 18:02:07.493595912 +0000 UTC m=+835.865384485" Jul 20 18:02:18.481226 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:02:18.481191 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-h76fd" Jul 20 18:03:12.040676 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:12.040598 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 18:03:12.041195 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:12.040924 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 18:03:24.669263 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:24.669231 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:03:44.603678 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:44.603644 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:03:55.808151 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.808118 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q"] Jul 20 18:03:55.811120 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.811098 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:55.813837 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.813820 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 20 18:03:55.813931 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.813820 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 20 18:03:55.813931 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.813888 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 20 18:03:55.814875 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.814851 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kpjkh\"" Jul 20 18:03:55.826683 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.826661 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q"] Jul 20 18:03:55.895111 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.895081 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wbr8q\" (UID: \"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:55.895111 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.895112 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wbr8q\" (UID: \"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:55.895317 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.895137 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wbr8q\" (UID: \"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:55.895317 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.895193 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wbr8q\" (UID: \"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:55.895317 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.895248 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfsq2\" (UniqueName: \"kubernetes.io/projected/3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3-kube-api-access-xfsq2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wbr8q\" (UID: \"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:55.895317 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.895294 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wbr8q\" (UID: \"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:55.996433 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.996390 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wbr8q\" (UID: \"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:55.996642 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.996454 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wbr8q\" (UID: \"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:55.996642 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.996472 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wbr8q\" (UID: \"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:55.996642 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.996495 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wbr8q\" (UID: \"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:55.996642 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.996516 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wbr8q\" (UID: \"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:55.996642 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.996546 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsq2\" (UniqueName: \"kubernetes.io/projected/3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3-kube-api-access-xfsq2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wbr8q\" (UID: \"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:55.996983 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.996962 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wbr8q\" (UID: \"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:55.997050 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.996961 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wbr8q\" (UID: \"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:55.997050 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.997034 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wbr8q\" (UID: \"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:55.998839 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.998820 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wbr8q\" (UID: \"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:55.999000 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:55.998984 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wbr8q\" (UID: \"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:56.005313 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:56.005288 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfsq2\" (UniqueName: \"kubernetes.io/projected/3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3-kube-api-access-xfsq2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wbr8q\" (UID: \"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:56.121614 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:56.121510 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:03:56.256800 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:56.256775 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q"] Jul 20 18:03:56.259356 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:03:56.259329 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b85d5e8_dd54_4ef0_ac16_d0dd6cdd79d3.slice/crio-a41fbf67c68779795a762ef9c522bc45aa2ee802c4e1c6757a79912c1b370665 WatchSource:0}: Error finding container a41fbf67c68779795a762ef9c522bc45aa2ee802c4e1c6757a79912c1b370665: Status 404 returned error can't find the container with id a41fbf67c68779795a762ef9c522bc45aa2ee802c4e1c6757a79912c1b370665 Jul 20 18:03:56.261156 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:56.261134 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 18:03:56.567414 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:56.567376 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:03:57.000724 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:57.000682 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" event={"ID":"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3","Type":"ContainerStarted","Data":"a41fbf67c68779795a762ef9c522bc45aa2ee802c4e1c6757a79912c1b370665"} Jul 20 18:03:58.007399 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:58.007362 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" event={"ID":"3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3","Type":"ContainerStarted","Data":"399a0095651d69b605beb7f82c5c587f5d138589545c416475c68a751fc2c4d4"} Jul 20 18:03:58.027336 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:03:58.027287 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" podStartSLOduration=1.981861093 podStartE2EDuration="3.02727078s" totalImagesPullingTime="1.045409687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:03:58.024911856 +0000 UTC m=+946.396700430" watchObservedRunningTime="2026-07-20 18:03:58.02727078 +0000 UTC m=+946.399059353" Jul 20 18:04:03.152991 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:04:03.152950 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:04:06.122365 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:04:06.122307 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:04:06.122874 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:04:06.122391 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:04:06.135348 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:04:06.135325 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:04:07.063534 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:04:07.063504 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wbr8q" Jul 20 18:04:16.556077 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:04:16.556048 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:04:24.856348 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:04:24.856308 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:04:36.558071 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:04:36.557997 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:06:00.768124 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:06:00.768081 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:06:11.244146 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:06:11.244064 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:06:21.246084 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:06:21.246045 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:08:04.552661 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:08:04.552630 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:08:12.072176 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:08:12.072144 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 18:08:12.073475 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:08:12.073449 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 18:08:12.448073 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:08:12.448039 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:08:44.260016 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:08:44.259982 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:08:53.850831 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:08:53.850786 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:09:02.357694 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:09:02.357659 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:09:11.247131 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:09:11.247099 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:09:21.851500 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:09:21.851463 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:09:32.210825 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:09:32.210796 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:09:42.953090 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:09:42.953055 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:09:52.651261 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:09:52.651223 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:10:15.556258 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:10:15.556225 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:10:25.360640 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:10:25.360603 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:10:48.945087 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:10:48.945003 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:10:58.951012 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:10:58.950976 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:11:13.449841 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:11:13.449809 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:11:22.449465 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:11:22.449431 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:11:33.358597 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:11:33.358556 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:11:42.646074 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:11:42.646041 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:11:52.048803 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:11:52.048762 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:13:12.103086 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:13:12.103059 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 18:13:12.106853 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:13:12.106828 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 18:13:39.644723 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:13:39.644630 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:13:50.146314 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:13:50.146275 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:14:00.655668 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:14:00.655631 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:14:12.256182 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:14:12.256142 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:14:22.144034 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:14:22.143998 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:14:34.176121 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:14:34.176090 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:14:41.945486 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:14:41.945447 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:14:53.048144 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:14:53.048111 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:15:04.547493 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:15:04.547411 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:15:13.152781 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:15:13.152741 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:15:24.464440 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:15:24.464404 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:15:45.553101 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:15:45.553065 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:15:55.458684 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:15:55.458651 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:16:02.688002 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.687961 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x"] Jul 20 18:16:02.690596 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.690579 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.693161 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.693139 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-b075c996-kserve-self-signed-certs\"" Jul 20 18:16:02.701910 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.701887 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x"] Jul 20 18:16:02.847179 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.847144 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mngdt\" (UniqueName: \"kubernetes.io/projected/b4aa5905-6ad8-4deb-a7ba-6186209c276d-kube-api-access-mngdt\") pod \"e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.847179 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.847183 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-tmp-dir\") pod \"e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.847420 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.847258 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-home\") pod \"e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.847420 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.847353 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4aa5905-6ad8-4deb-a7ba-6186209c276d-tls-certs\") pod \"e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.847420 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.847392 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-dshm\") pod \"e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.847545 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.847423 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-model-cache\") pod \"e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.948643 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.948554 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-model-cache\") pod \"e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.948643 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.948598 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mngdt\" (UniqueName: \"kubernetes.io/projected/b4aa5905-6ad8-4deb-a7ba-6186209c276d-kube-api-access-mngdt\") pod \"e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.948643 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.948619 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-tmp-dir\") pod \"e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.948902 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.948755 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-home\") pod \"e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.948902 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.948812 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4aa5905-6ad8-4deb-a7ba-6186209c276d-tls-certs\") pod \"e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.948902 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.948856 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-dshm\") pod \"e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.949063 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.949032 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-tmp-dir\") pod \"e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.949279 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.949258 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-model-cache\") pod \"e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.949684 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.949650 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-home\") pod \"e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.955760 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.952130 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-dshm\") pod \"e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.955760 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.952454 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4aa5905-6ad8-4deb-a7ba-6186209c276d-tls-certs\") pod \"e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.958203 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.958181 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mngdt\" (UniqueName: \"kubernetes.io/projected/b4aa5905-6ad8-4deb-a7ba-6186209c276d-kube-api-access-mngdt\") pod \"e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:02.999222 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:02.999179 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:03.133117 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:03.130976 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x"] Jul 20 18:16:03.133587 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:16:03.133558 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4aa5905_6ad8_4deb_a7ba_6186209c276d.slice/crio-fdb76b349aebf0d6f149db1400dc31910bc1477fefa456cbc72d613a51f87090 WatchSource:0}: Error finding container fdb76b349aebf0d6f149db1400dc31910bc1477fefa456cbc72d613a51f87090: Status 404 returned error can't find the container with id fdb76b349aebf0d6f149db1400dc31910bc1477fefa456cbc72d613a51f87090 Jul 20 18:16:03.568556 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:03.568521 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" event={"ID":"b4aa5905-6ad8-4deb-a7ba-6186209c276d","Type":"ContainerStarted","Data":"f043345174411a4a872a5c614e6dc85098ddaddf38af382dbe3501bc7409a3df"} Jul 20 18:16:03.568556 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:03.568563 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" event={"ID":"b4aa5905-6ad8-4deb-a7ba-6186209c276d","Type":"ContainerStarted","Data":"fdb76b349aebf0d6f149db1400dc31910bc1477fefa456cbc72d613a51f87090"} Jul 20 18:16:03.602313 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:03.602264 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" podStartSLOduration=1.602247237 podStartE2EDuration="1.602247237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:16:03.600296342 +0000 UTC m=+1671.972084917" watchObservedRunningTime="2026-07-20 18:16:03.602247237 +0000 UTC m=+1671.974035811" Jul 20 18:16:09.355760 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:09.354506 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:16:12.999857 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:12.999820 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:13.000300 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:12.999872 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:13.012589 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:13.012558 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:13.627350 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:13.627320 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:35.613591 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:35.613505 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x"] Jul 20 18:16:38.248364 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:38.248331 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:16:50.634403 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:50.634359 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" podUID="b4aa5905-6ad8-4deb-a7ba-6186209c276d" containerName="main" containerID="cri-o://f043345174411a4a872a5c614e6dc85098ddaddf38af382dbe3501bc7409a3df" gracePeriod=45 Jul 20 18:16:50.798003 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:50.797974 2583 generic.go:356] "Generic (PLEG): container finished" podID="b4aa5905-6ad8-4deb-a7ba-6186209c276d" containerID="f043345174411a4a872a5c614e6dc85098ddaddf38af382dbe3501bc7409a3df" exitCode=0 Jul 20 18:16:50.798171 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:50.798045 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" event={"ID":"b4aa5905-6ad8-4deb-a7ba-6186209c276d","Type":"ContainerDied","Data":"f043345174411a4a872a5c614e6dc85098ddaddf38af382dbe3501bc7409a3df"} Jul 20 18:16:50.876012 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:50.875987 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:50.951431 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:50.951391 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:16:50.958147 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:50.958119 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4aa5905-6ad8-4deb-a7ba-6186209c276d-tls-certs\") pod \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " Jul 20 18:16:50.958269 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:50.958161 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mngdt\" (UniqueName: \"kubernetes.io/projected/b4aa5905-6ad8-4deb-a7ba-6186209c276d-kube-api-access-mngdt\") pod \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " Jul 20 18:16:50.958269 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:50.958182 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-model-cache\") pod \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " Jul 20 18:16:50.958269 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:50.958258 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-tmp-dir\") pod \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " Jul 20 18:16:50.958400 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:50.958298 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-home\") pod \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " Jul 20 18:16:50.958400 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:50.958319 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-dshm\") pod \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\" (UID: \"b4aa5905-6ad8-4deb-a7ba-6186209c276d\") " Jul 20 18:16:50.958550 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:50.958518 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4aa5905-6ad8-4deb-a7ba-6186209c276d" (UID: "b4aa5905-6ad8-4deb-a7ba-6186209c276d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:16:50.958653 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:50.958560 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4aa5905-6ad8-4deb-a7ba-6186209c276d" (UID: "b4aa5905-6ad8-4deb-a7ba-6186209c276d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:16:50.958653 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:50.958612 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-home" (OuterVolumeSpecName: "home") pod "b4aa5905-6ad8-4deb-a7ba-6186209c276d" (UID: "b4aa5905-6ad8-4deb-a7ba-6186209c276d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:16:50.960380 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:50.960351 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-dshm" (OuterVolumeSpecName: "dshm") pod "b4aa5905-6ad8-4deb-a7ba-6186209c276d" (UID: "b4aa5905-6ad8-4deb-a7ba-6186209c276d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:16:50.960459 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:50.960402 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4aa5905-6ad8-4deb-a7ba-6186209c276d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4aa5905-6ad8-4deb-a7ba-6186209c276d" (UID: "b4aa5905-6ad8-4deb-a7ba-6186209c276d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 18:16:50.960459 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:50.960444 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4aa5905-6ad8-4deb-a7ba-6186209c276d-kube-api-access-mngdt" (OuterVolumeSpecName: "kube-api-access-mngdt") pod "b4aa5905-6ad8-4deb-a7ba-6186209c276d" (UID: "b4aa5905-6ad8-4deb-a7ba-6186209c276d"). InnerVolumeSpecName "kube-api-access-mngdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:16:51.059206 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:51.059170 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-home\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:16:51.059206 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:51.059198 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-dshm\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:16:51.059206 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:51.059207 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4aa5905-6ad8-4deb-a7ba-6186209c276d-tls-certs\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:16:51.059206 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:51.059218 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mngdt\" (UniqueName: \"kubernetes.io/projected/b4aa5905-6ad8-4deb-a7ba-6186209c276d-kube-api-access-mngdt\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:16:51.059463 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:51.059226 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-model-cache\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:16:51.059463 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:51.059235 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4aa5905-6ad8-4deb-a7ba-6186209c276d-tmp-dir\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:16:51.804459 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:51.804430 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" Jul 20 18:16:51.804459 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:51.804447 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x" event={"ID":"b4aa5905-6ad8-4deb-a7ba-6186209c276d","Type":"ContainerDied","Data":"fdb76b349aebf0d6f149db1400dc31910bc1477fefa456cbc72d613a51f87090"} Jul 20 18:16:51.804951 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:51.804496 2583 scope.go:117] "RemoveContainer" containerID="f043345174411a4a872a5c614e6dc85098ddaddf38af382dbe3501bc7409a3df" Jul 20 18:16:51.825706 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:51.825681 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x"] Jul 20 18:16:51.829693 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:51.829674 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-b075c996-kserve-5d945946f5-dvq9x"] Jul 20 18:16:52.169864 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:52.169788 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4aa5905-6ad8-4deb-a7ba-6186209c276d" path="/var/lib/kubelet/pods/b4aa5905-6ad8-4deb-a7ba-6186209c276d/volumes" Jul 20 18:16:58.050112 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:16:58.050071 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:17:13.249282 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:17:13.249249 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:17:24.739427 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:17:24.739392 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:17:33.849468 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:17:33.849430 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:17:44.645585 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:17:44.645550 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:17:53.648557 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:17:53.648520 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:18:02.757343 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:18:02.757260 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:18:12.133413 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:18:12.133387 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 18:18:12.139906 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:18:12.139881 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 18:18:15.556304 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:18:15.556270 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:18:25.561190 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:18:25.561154 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:18:35.359720 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:18:35.359684 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:18:48.048402 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:18:48.048363 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:18:59.345636 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:18:59.345599 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:19:07.646460 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:19:07.646425 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:19:19.049846 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:19:19.049810 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:19:31.138117 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:19:31.138087 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:19:43.557514 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:19:43.557432 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:19:55.598438 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:19:55.598408 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:20:07.759704 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:07.759668 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:20:23.349368 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:23.349335 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:20:55.649944 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:55.649910 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:20:58.587511 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.587476 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh"] Jul 20 18:20:58.587952 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.587818 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4aa5905-6ad8-4deb-a7ba-6186209c276d" containerName="main" Jul 20 18:20:58.587952 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.587831 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4aa5905-6ad8-4deb-a7ba-6186209c276d" containerName="main" Jul 20 18:20:58.587952 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.587892 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4aa5905-6ad8-4deb-a7ba-6186209c276d" containerName="main" Jul 20 18:20:58.590973 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.590951 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.593645 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.593624 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-34379a7f-gw-openshift-default-dockercfg-twfqc\"" Jul 20 18:20:58.606923 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.606897 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh"] Jul 20 18:20:58.672421 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.672384 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-workload-certs\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.672421 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.672425 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istiod-ca-cert\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.672631 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.672483 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-credential-socket\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.672631 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.672517 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-token\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.672631 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.672585 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-workload-socket\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.672631 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.672608 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-podinfo\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.672782 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.672639 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-envoy\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.672782 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.672674 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-data\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.672782 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.672694 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x79tv\" (UniqueName: \"kubernetes.io/projected/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-kube-api-access-x79tv\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.773523 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.773476 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-workload-socket\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.773523 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.773524 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-podinfo\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.773843 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.773562 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-envoy\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.773843 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.773595 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-data\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.773843 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.773618 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x79tv\" (UniqueName: \"kubernetes.io/projected/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-kube-api-access-x79tv\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.773843 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.773672 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-workload-certs\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.773843 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.773698 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istiod-ca-cert\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.773843 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.773763 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-credential-socket\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.773843 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.773789 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-token\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.774193 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.773974 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-workload-socket\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.774193 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.774042 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-data\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.774305 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.774226 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-workload-certs\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.774362 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.774334 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-credential-socket\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.774682 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.774656 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istiod-ca-cert\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.775917 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.775886 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-envoy\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.776170 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.776151 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-podinfo\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.783269 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.783247 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-token\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.783894 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.783873 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x79tv\" (UniqueName: \"kubernetes.io/projected/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-kube-api-access-x79tv\") pod \"e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:58.900752 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:58.900658 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:20:59.034074 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:59.034046 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh"] Jul 20 18:20:59.037677 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:20:59.037638 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89dfad2a_ba8f_4009_b96a_2cf9d1056e42.slice/crio-e336218d875e9bd3ac71d50f3f2e839e5ce705bcb7cd68666db648d77a3b6d0e WatchSource:0}: Error finding container e336218d875e9bd3ac71d50f3f2e839e5ce705bcb7cd68666db648d77a3b6d0e: Status 404 returned error can't find the container with id e336218d875e9bd3ac71d50f3f2e839e5ce705bcb7cd68666db648d77a3b6d0e Jul 20 18:20:59.039802 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:59.039784 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 18:20:59.990143 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:20:59.990097 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" event={"ID":"89dfad2a-ba8f-4009-b96a-2cf9d1056e42","Type":"ContainerStarted","Data":"e336218d875e9bd3ac71d50f3f2e839e5ce705bcb7cd68666db648d77a3b6d0e"} Jul 20 18:21:01.697479 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:01.697444 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:21:01.697771 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:01.697515 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:21:01.697771 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:01.697545 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:21:02.005195 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:02.005091 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" event={"ID":"89dfad2a-ba8f-4009-b96a-2cf9d1056e42","Type":"ContainerStarted","Data":"0d416b5212bc2b4fe7c45029827a747d6b2b24acc8dd6fb1d87ea6c25e28694d"} Jul 20 18:21:02.028767 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:02.028701 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" podStartSLOduration=1.371504655 podStartE2EDuration="4.028686918s" totalImagesPullingTime="2.657182263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:21:02.026426364 +0000 UTC m=+1970.398214944" watchObservedRunningTime="2026-07-20 18:21:02.028686918 +0000 UTC m=+1970.400475492" Jul 20 18:21:02.900943 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:02.900861 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:21:02.905334 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:02.905310 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:21:03.010175 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:03.010146 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:21:03.011005 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:03.010980 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:21:07.652760 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:07.652706 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:21:16.076196 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:16.076162 2583 generic.go:356] "Generic (PLEG): container finished" podID="89dfad2a-ba8f-4009-b96a-2cf9d1056e42" containerID="0d416b5212bc2b4fe7c45029827a747d6b2b24acc8dd6fb1d87ea6c25e28694d" exitCode=0 Jul 20 18:21:16.076636 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:16.076237 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" event={"ID":"89dfad2a-ba8f-4009-b96a-2cf9d1056e42","Type":"ContainerDied","Data":"0d416b5212bc2b4fe7c45029827a747d6b2b24acc8dd6fb1d87ea6c25e28694d"} Jul 20 18:21:16.076765 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:16.076723 2583 scope.go:117] "RemoveContainer" containerID="0d416b5212bc2b4fe7c45029827a747d6b2b24acc8dd6fb1d87ea6c25e28694d" Jul 20 18:21:16.077832 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:16.077802 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:21:16.077910 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:16.077868 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:21:16.077910 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:16.077897 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:21:17.083031 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:17.082998 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" event={"ID":"89dfad2a-ba8f-4009-b96a-2cf9d1056e42","Type":"ContainerStarted","Data":"6a68512deae9099577347ee80eae44a4f426fba0e2b9588af2719ef5e93d3aa7"} Jul 20 18:21:17.901142 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:17.901104 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:21:17.902270 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:17.902245 2583 patch_prober.go:28] interesting pod/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 20 18:21:17.902393 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:17.902294 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" podUID="89dfad2a-ba8f-4009-b96a-2cf9d1056e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 20 18:21:18.010858 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:18.010827 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:21:18.901423 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:18.901386 2583 patch_prober.go:28] interesting pod/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 20 18:21:18.901855 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:18.901442 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" podUID="89dfad2a-ba8f-4009-b96a-2cf9d1056e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 20 18:21:19.901530 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:19.901494 2583 patch_prober.go:28] interesting pod/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 20 18:21:19.901954 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:19.901567 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" podUID="89dfad2a-ba8f-4009-b96a-2cf9d1056e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 20 18:21:20.901942 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:20.901909 2583 patch_prober.go:28] interesting pod/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 20 18:21:20.902351 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:20.901963 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" podUID="89dfad2a-ba8f-4009-b96a-2cf9d1056e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 20 18:21:21.905410 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:21.905372 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:21:21.906427 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:21.906402 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:21:42.050443 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:42.050399 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:21:43.817095 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:43.817059 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh"] Jul 20 18:21:43.817504 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:43.817262 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" podUID="89dfad2a-ba8f-4009-b96a-2cf9d1056e42" containerName="istio-proxy" containerID="cri-o://6a68512deae9099577347ee80eae44a4f426fba0e2b9588af2719ef5e93d3aa7" gracePeriod=30 Jul 20 18:21:44.658256 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.658222 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc"] Jul 20 18:21:44.661984 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.661966 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.665240 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.665212 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bd66d6f4-openshift-default-dockercfg-qc6n4\"" Jul 20 18:21:44.672708 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.672687 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc"] Jul 20 18:21:44.772120 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.772087 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-workload-certs\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.772440 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.772417 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-credential-socket\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.772662 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.772631 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-envoy\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.772826 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.772812 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istiod-ca-cert\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.772988 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.772968 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-workload-socket\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.773109 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.773097 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-podinfo\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.773243 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.773230 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-token\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.773374 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.773360 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6ckb\" (UniqueName: \"kubernetes.io/projected/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-kube-api-access-n6ckb\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.773514 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.773501 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-data\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.874264 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.874237 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-workload-certs\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.874264 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.874266 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-credential-socket\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.874784 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.874293 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-envoy\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.874784 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.874397 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istiod-ca-cert\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.874784 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.874444 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-workload-socket\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.874784 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.874463 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-podinfo\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.874784 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.874482 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-token\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.874784 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.874499 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6ckb\" (UniqueName: \"kubernetes.io/projected/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-kube-api-access-n6ckb\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.874784 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.874676 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-data\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.874784 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.874695 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-workload-certs\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.875138 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.874647 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-credential-socket\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.875138 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.874978 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-workload-socket\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.875138 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.875013 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-data\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.875283 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.875238 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istiod-ca-cert\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.877547 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.877524 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-podinfo\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.877673 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.877527 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-envoy\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.884225 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.884197 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6ckb\" (UniqueName: \"kubernetes.io/projected/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-kube-api-access-n6ckb\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.884225 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.884215 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-token\") pod \"e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:44.973401 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:44.973369 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:45.113622 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:45.113589 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc"] Jul 20 18:21:45.115283 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:21:45.115257 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32c1bca_620e_42ab_a9bc_d7d6788c4f9f.slice/crio-77fa68e37a70c2eb964c7bdc4f79e6e74d4332d7321225ed8351133678d624fa WatchSource:0}: Error finding container 77fa68e37a70c2eb964c7bdc4f79e6e74d4332d7321225ed8351133678d624fa: Status 404 returned error can't find the container with id 77fa68e37a70c2eb964c7bdc4f79e6e74d4332d7321225ed8351133678d624fa Jul 20 18:21:45.117573 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:45.117535 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:21:45.117651 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:45.117618 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:21:45.117706 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:45.117658 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:21:45.229852 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:45.229760 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" event={"ID":"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f","Type":"ContainerStarted","Data":"e3f7276f3a449e39d278bda7e145f42cfca0ee615e1d0c4514a501ac42fa492e"} Jul 20 18:21:45.230018 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:45.229859 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" event={"ID":"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f","Type":"ContainerStarted","Data":"77fa68e37a70c2eb964c7bdc4f79e6e74d4332d7321225ed8351133678d624fa"} Jul 20 18:21:45.267489 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:45.267428 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" podStartSLOduration=1.267405112 podStartE2EDuration="1.267405112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:21:45.265091145 +0000 UTC m=+2013.636879720" watchObservedRunningTime="2026-07-20 18:21:45.267405112 +0000 UTC m=+2013.639193668" Jul 20 18:21:45.974398 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:45.974359 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:46.979758 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:46.979704 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:47.239698 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:47.239618 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:47.240667 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:47.240650 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:21:49.067941 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.067914 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:21:49.207573 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.207546 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-data\") pod \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " Jul 20 18:21:49.207722 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.207594 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-envoy\") pod \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " Jul 20 18:21:49.207722 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.207629 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-credential-socket\") pod \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " Jul 20 18:21:49.207722 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.207655 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-workload-certs\") pod \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " Jul 20 18:21:49.207722 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.207672 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x79tv\" (UniqueName: \"kubernetes.io/projected/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-kube-api-access-x79tv\") pod \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " Jul 20 18:21:49.207722 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.207701 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-token\") pod \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " Jul 20 18:21:49.208053 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.207723 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-workload-socket\") pod \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " Jul 20 18:21:49.208053 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.207783 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istiod-ca-cert\") pod \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " Jul 20 18:21:49.208053 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.207815 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-podinfo\") pod \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\" (UID: \"89dfad2a-ba8f-4009-b96a-2cf9d1056e42\") " Jul 20 18:21:49.208053 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.208010 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "89dfad2a-ba8f-4009-b96a-2cf9d1056e42" (UID: "89dfad2a-ba8f-4009-b96a-2cf9d1056e42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:21:49.208255 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.208139 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-credential-socket\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:21:49.208255 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.208138 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "89dfad2a-ba8f-4009-b96a-2cf9d1056e42" (UID: "89dfad2a-ba8f-4009-b96a-2cf9d1056e42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:21:49.208255 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.208022 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "89dfad2a-ba8f-4009-b96a-2cf9d1056e42" (UID: "89dfad2a-ba8f-4009-b96a-2cf9d1056e42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:21:49.208408 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.208342 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "89dfad2a-ba8f-4009-b96a-2cf9d1056e42" (UID: "89dfad2a-ba8f-4009-b96a-2cf9d1056e42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 18:21:49.208464 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.208395 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-data" (OuterVolumeSpecName: "istio-data") pod "89dfad2a-ba8f-4009-b96a-2cf9d1056e42" (UID: "89dfad2a-ba8f-4009-b96a-2cf9d1056e42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:21:49.210059 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.210034 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "89dfad2a-ba8f-4009-b96a-2cf9d1056e42" (UID: "89dfad2a-ba8f-4009-b96a-2cf9d1056e42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:21:49.210184 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.210167 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-kube-api-access-x79tv" (OuterVolumeSpecName: "kube-api-access-x79tv") pod "89dfad2a-ba8f-4009-b96a-2cf9d1056e42" (UID: "89dfad2a-ba8f-4009-b96a-2cf9d1056e42"). InnerVolumeSpecName "kube-api-access-x79tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:21:49.210240 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.210180 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "89dfad2a-ba8f-4009-b96a-2cf9d1056e42" (UID: "89dfad2a-ba8f-4009-b96a-2cf9d1056e42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 18:21:49.210240 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.210198 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-token" (OuterVolumeSpecName: "istio-token") pod "89dfad2a-ba8f-4009-b96a-2cf9d1056e42" (UID: "89dfad2a-ba8f-4009-b96a-2cf9d1056e42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:21:49.251611 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.251580 2583 generic.go:356] "Generic (PLEG): container finished" podID="89dfad2a-ba8f-4009-b96a-2cf9d1056e42" containerID="6a68512deae9099577347ee80eae44a4f426fba0e2b9588af2719ef5e93d3aa7" exitCode=0 Jul 20 18:21:49.251783 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.251626 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" Jul 20 18:21:49.251783 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.251665 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" event={"ID":"89dfad2a-ba8f-4009-b96a-2cf9d1056e42","Type":"ContainerDied","Data":"6a68512deae9099577347ee80eae44a4f426fba0e2b9588af2719ef5e93d3aa7"} Jul 20 18:21:49.251783 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.251721 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh" event={"ID":"89dfad2a-ba8f-4009-b96a-2cf9d1056e42","Type":"ContainerDied","Data":"e336218d875e9bd3ac71d50f3f2e839e5ce705bcb7cd68666db648d77a3b6d0e"} Jul 20 18:21:49.251783 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.251782 2583 scope.go:117] "RemoveContainer" containerID="6a68512deae9099577347ee80eae44a4f426fba0e2b9588af2719ef5e93d3aa7" Jul 20 18:21:49.267535 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.267514 2583 scope.go:117] "RemoveContainer" containerID="0d416b5212bc2b4fe7c45029827a747d6b2b24acc8dd6fb1d87ea6c25e28694d" Jul 20 18:21:49.275801 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.275783 2583 scope.go:117] "RemoveContainer" containerID="6a68512deae9099577347ee80eae44a4f426fba0e2b9588af2719ef5e93d3aa7" Jul 20 18:21:49.276052 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:21:49.276031 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a68512deae9099577347ee80eae44a4f426fba0e2b9588af2719ef5e93d3aa7\": container with ID starting with 6a68512deae9099577347ee80eae44a4f426fba0e2b9588af2719ef5e93d3aa7 not found: ID does not exist" containerID="6a68512deae9099577347ee80eae44a4f426fba0e2b9588af2719ef5e93d3aa7" Jul 20 18:21:49.276118 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.276059 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a68512deae9099577347ee80eae44a4f426fba0e2b9588af2719ef5e93d3aa7"} err="failed to get container status \"6a68512deae9099577347ee80eae44a4f426fba0e2b9588af2719ef5e93d3aa7\": rpc error: code = NotFound desc = could not find container \"6a68512deae9099577347ee80eae44a4f426fba0e2b9588af2719ef5e93d3aa7\": container with ID starting with 6a68512deae9099577347ee80eae44a4f426fba0e2b9588af2719ef5e93d3aa7 not found: ID does not exist" Jul 20 18:21:49.276118 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.276075 2583 scope.go:117] "RemoveContainer" containerID="0d416b5212bc2b4fe7c45029827a747d6b2b24acc8dd6fb1d87ea6c25e28694d" Jul 20 18:21:49.276310 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:21:49.276295 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d416b5212bc2b4fe7c45029827a747d6b2b24acc8dd6fb1d87ea6c25e28694d\": container with ID starting with 0d416b5212bc2b4fe7c45029827a747d6b2b24acc8dd6fb1d87ea6c25e28694d not found: ID does not exist" containerID="0d416b5212bc2b4fe7c45029827a747d6b2b24acc8dd6fb1d87ea6c25e28694d" Jul 20 18:21:49.276349 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.276313 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d416b5212bc2b4fe7c45029827a747d6b2b24acc8dd6fb1d87ea6c25e28694d"} err="failed to get container status \"0d416b5212bc2b4fe7c45029827a747d6b2b24acc8dd6fb1d87ea6c25e28694d\": rpc error: code = NotFound desc = could not find container \"0d416b5212bc2b4fe7c45029827a747d6b2b24acc8dd6fb1d87ea6c25e28694d\": container with ID starting with 0d416b5212bc2b4fe7c45029827a747d6b2b24acc8dd6fb1d87ea6c25e28694d not found: ID does not exist" Jul 20 18:21:49.281184 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.281159 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh"] Jul 20 18:21:49.283207 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.283177 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-34379a7f-gw-openshift-default-54bb846b7f-2q7vh"] Jul 20 18:21:49.309337 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.309318 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istiod-ca-cert\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:21:49.309337 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.309338 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-podinfo\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:21:49.309464 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.309346 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-data\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:21:49.309464 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.309356 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-envoy\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:21:49.309464 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.309364 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-workload-certs\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:21:49.309464 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.309372 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x79tv\" (UniqueName: \"kubernetes.io/projected/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-kube-api-access-x79tv\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:21:49.309464 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.309381 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-istio-token\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:21:49.309464 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:49.309388 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dfad2a-ba8f-4009-b96a-2cf9d1056e42-workload-socket\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:21:50.170379 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:50.170346 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89dfad2a-ba8f-4009-b96a-2cf9d1056e42" path="/var/lib/kubelet/pods/89dfad2a-ba8f-4009-b96a-2cf9d1056e42/volumes" Jul 20 18:21:52.958869 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:21:52.958834 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:22:01.316873 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:01.316843 2583 generic.go:356] "Generic (PLEG): container finished" podID="e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" containerID="e3f7276f3a449e39d278bda7e145f42cfca0ee615e1d0c4514a501ac42fa492e" exitCode=0 Jul 20 18:22:01.317258 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:01.316879 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" event={"ID":"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f","Type":"ContainerDied","Data":"e3f7276f3a449e39d278bda7e145f42cfca0ee615e1d0c4514a501ac42fa492e"} Jul 20 18:22:01.317258 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:01.317229 2583 scope.go:117] "RemoveContainer" containerID="e3f7276f3a449e39d278bda7e145f42cfca0ee615e1d0c4514a501ac42fa492e" Jul 20 18:22:01.318201 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:01.318170 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:22:01.318275 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:01.318257 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:22:01.318320 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:01.318306 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:22:02.240103 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:02.240060 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:22:02.327429 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:02.327375 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" event={"ID":"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f","Type":"ContainerStarted","Data":"84def4eb0290af57a8f6c7367210ba12d91ae5bd491894e3ec67d5e17841776d"} Jul 20 18:22:02.973821 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:02.973775 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:22:02.974876 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:02.974854 2583 patch_prober.go:28] interesting pod/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 20 18:22:02.974933 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:02.974918 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" podUID="e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 20 18:22:03.973958 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:03.973922 2583 patch_prober.go:28] interesting pod/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 20 18:22:03.974354 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:03.973990 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" podUID="e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 20 18:22:04.974231 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:04.974190 2583 patch_prober.go:28] interesting pod/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 20 18:22:04.974708 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:04.974245 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" podUID="e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 20 18:22:05.974621 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:05.974582 2583 patch_prober.go:28] interesting pod/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 20 18:22:05.975067 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:05.974648 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" podUID="e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 20 18:22:06.978421 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:06.978388 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:22:06.978942 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:06.978778 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:22:06.979509 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:06.979493 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:22:25.849632 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:25.849597 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:22:28.748129 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:28.748096 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc"] Jul 20 18:22:28.748495 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:28.748338 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" podUID="e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" containerName="istio-proxy" containerID="cri-o://84def4eb0290af57a8f6c7367210ba12d91ae5bd491894e3ec67d5e17841776d" gracePeriod=30 Jul 20 18:22:29.509655 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.509623 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd"] Jul 20 18:22:29.509998 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.509985 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89dfad2a-ba8f-4009-b96a-2cf9d1056e42" containerName="istio-proxy" Jul 20 18:22:29.510044 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.510000 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="89dfad2a-ba8f-4009-b96a-2cf9d1056e42" containerName="istio-proxy" Jul 20 18:22:29.510044 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.510009 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89dfad2a-ba8f-4009-b96a-2cf9d1056e42" containerName="istio-proxy" Jul 20 18:22:29.510044 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.510014 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="89dfad2a-ba8f-4009-b96a-2cf9d1056e42" containerName="istio-proxy" Jul 20 18:22:29.510134 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.510062 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="89dfad2a-ba8f-4009-b96a-2cf9d1056e42" containerName="istio-proxy" Jul 20 18:22:29.510134 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.510070 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="89dfad2a-ba8f-4009-b96a-2cf9d1056e42" containerName="istio-proxy" Jul 20 18:22:29.513181 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.513165 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.515769 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.515723 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e0a9248b-openshift-default-dockercfg-62w6l\"" Jul 20 18:22:29.530431 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.530404 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd"] Jul 20 18:22:29.554901 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.554867 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istiod-ca-cert\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.555036 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.554908 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-workload-certs\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.555036 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.554959 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-token\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.555036 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.555029 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-credential-socket\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.555149 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.555050 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-podinfo\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.555149 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.555098 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-envoy\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.555149 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.555118 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-data\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.555149 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.555138 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brkwt\" (UniqueName: \"kubernetes.io/projected/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-kube-api-access-brkwt\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.555293 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.555207 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-workload-socket\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.656582 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.656549 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-workload-socket\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.656810 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.656609 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istiod-ca-cert\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.656810 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.656643 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-workload-certs\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.656810 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.656754 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-token\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.656987 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.656838 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-credential-socket\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.656987 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.656866 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-podinfo\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.656987 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.656905 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-envoy\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.656987 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.656931 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-data\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.656987 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.656961 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brkwt\" (UniqueName: \"kubernetes.io/projected/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-kube-api-access-brkwt\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.657239 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.657034 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-workload-socket\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.657239 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.657232 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-credential-socket\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.657344 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.657281 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-data\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.657456 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.657436 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istiod-ca-cert\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.657521 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.657439 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-workload-certs\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.659328 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.659263 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-envoy\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.659748 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.659703 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-podinfo\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.668913 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.668892 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-token\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.669109 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.669093 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brkwt\" (UniqueName: \"kubernetes.io/projected/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-kube-api-access-brkwt\") pod \"e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.823334 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.823259 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:29.961711 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.961683 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd"] Jul 20 18:22:29.962850 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:22:29.962829 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d297c0_bdaa_4d74_9eed_6c4f922aeb62.slice/crio-ed82a912008e36555a08f83704bb8b01952ba5b11e3fcf8b544ed21ba60874aa WatchSource:0}: Error finding container ed82a912008e36555a08f83704bb8b01952ba5b11e3fcf8b544ed21ba60874aa: Status 404 returned error can't find the container with id ed82a912008e36555a08f83704bb8b01952ba5b11e3fcf8b544ed21ba60874aa Jul 20 18:22:29.964819 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.964789 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:22:29.964902 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.964850 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:22:29.964902 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:29.964879 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:22:30.469743 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:30.469692 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" event={"ID":"15d297c0-bdaa-4d74-9eed-6c4f922aeb62","Type":"ContainerStarted","Data":"faece67d3baf471ef06a91c563805d04d36503fa27829103a03f0582515ac734"} Jul 20 18:22:30.469953 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:30.469761 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" event={"ID":"15d297c0-bdaa-4d74-9eed-6c4f922aeb62","Type":"ContainerStarted","Data":"ed82a912008e36555a08f83704bb8b01952ba5b11e3fcf8b544ed21ba60874aa"} Jul 20 18:22:30.499631 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:30.499553 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" podStartSLOduration=1.499532681 podStartE2EDuration="1.499532681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:22:30.496717867 +0000 UTC m=+2058.868506435" watchObservedRunningTime="2026-07-20 18:22:30.499532681 +0000 UTC m=+2058.871321258" Jul 20 18:22:30.824062 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:30.823959 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:31.828414 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:31.828379 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:32.479935 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:32.479843 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:32.480829 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:32.480805 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:34.002950 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.002925 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:22:34.096879 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.096690 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6ckb\" (UniqueName: \"kubernetes.io/projected/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-kube-api-access-n6ckb\") pod \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " Jul 20 18:22:34.096879 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.096751 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-token\") pod \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " Jul 20 18:22:34.096879 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.096795 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-workload-certs\") pod \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " Jul 20 18:22:34.096879 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.096837 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-workload-socket\") pod \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " Jul 20 18:22:34.097221 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.096961 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-data\") pod \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " Jul 20 18:22:34.097221 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.097005 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-envoy\") pod \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " Jul 20 18:22:34.097221 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.097062 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istiod-ca-cert\") pod \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " Jul 20 18:22:34.097221 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.097094 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-credential-socket\") pod \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " Jul 20 18:22:34.097221 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.097111 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" (UID: "e32c1bca-620e-42ab-a9bc-d7d6788c4f9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:22:34.097221 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.097122 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" (UID: "e32c1bca-620e-42ab-a9bc-d7d6788c4f9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:22:34.097221 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.097120 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-podinfo\") pod \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\" (UID: \"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f\") " Jul 20 18:22:34.097581 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.097482 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" (UID: "e32c1bca-620e-42ab-a9bc-d7d6788c4f9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 18:22:34.097581 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.097469 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" (UID: "e32c1bca-620e-42ab-a9bc-d7d6788c4f9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:22:34.097717 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.097697 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-workload-certs\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:22:34.097811 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.097720 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-workload-socket\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:22:34.097811 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.097753 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istiod-ca-cert\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:22:34.097811 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.097762 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-credential-socket\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:22:34.098270 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.098242 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-data" (OuterVolumeSpecName: "istio-data") pod "e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" (UID: "e32c1bca-620e-42ab-a9bc-d7d6788c4f9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:22:34.099141 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.099112 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-kube-api-access-n6ckb" (OuterVolumeSpecName: "kube-api-access-n6ckb") pod "e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" (UID: "e32c1bca-620e-42ab-a9bc-d7d6788c4f9f"). InnerVolumeSpecName "kube-api-access-n6ckb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:22:34.099141 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.099123 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-token" (OuterVolumeSpecName: "istio-token") pod "e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" (UID: "e32c1bca-620e-42ab-a9bc-d7d6788c4f9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:22:34.099432 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.099415 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" (UID: "e32c1bca-620e-42ab-a9bc-d7d6788c4f9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 18:22:34.099513 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.099499 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" (UID: "e32c1bca-620e-42ab-a9bc-d7d6788c4f9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:22:34.198447 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.198409 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6ckb\" (UniqueName: \"kubernetes.io/projected/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-kube-api-access-n6ckb\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:22:34.198447 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.198442 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-token\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:22:34.198640 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.198458 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-data\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:22:34.198640 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.198467 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-envoy\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:22:34.198640 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.198474 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f-istio-podinfo\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:22:34.490854 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.490820 2583 generic.go:356] "Generic (PLEG): container finished" podID="e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" containerID="84def4eb0290af57a8f6c7367210ba12d91ae5bd491894e3ec67d5e17841776d" exitCode=0 Jul 20 18:22:34.491058 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.490869 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" Jul 20 18:22:34.491058 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.490909 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" event={"ID":"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f","Type":"ContainerDied","Data":"84def4eb0290af57a8f6c7367210ba12d91ae5bd491894e3ec67d5e17841776d"} Jul 20 18:22:34.491058 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.490972 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc" event={"ID":"e32c1bca-620e-42ab-a9bc-d7d6788c4f9f","Type":"ContainerDied","Data":"77fa68e37a70c2eb964c7bdc4f79e6e74d4332d7321225ed8351133678d624fa"} Jul 20 18:22:34.491058 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.491005 2583 scope.go:117] "RemoveContainer" containerID="84def4eb0290af57a8f6c7367210ba12d91ae5bd491894e3ec67d5e17841776d" Jul 20 18:22:34.499846 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.499828 2583 scope.go:117] "RemoveContainer" containerID="e3f7276f3a449e39d278bda7e145f42cfca0ee615e1d0c4514a501ac42fa492e" Jul 20 18:22:34.507997 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.507953 2583 scope.go:117] "RemoveContainer" containerID="84def4eb0290af57a8f6c7367210ba12d91ae5bd491894e3ec67d5e17841776d" Jul 20 18:22:34.508383 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:22:34.508348 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84def4eb0290af57a8f6c7367210ba12d91ae5bd491894e3ec67d5e17841776d\": container with ID starting with 84def4eb0290af57a8f6c7367210ba12d91ae5bd491894e3ec67d5e17841776d not found: ID does not exist" containerID="84def4eb0290af57a8f6c7367210ba12d91ae5bd491894e3ec67d5e17841776d" Jul 20 18:22:34.508514 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.508390 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84def4eb0290af57a8f6c7367210ba12d91ae5bd491894e3ec67d5e17841776d"} err="failed to get container status \"84def4eb0290af57a8f6c7367210ba12d91ae5bd491894e3ec67d5e17841776d\": rpc error: code = NotFound desc = could not find container \"84def4eb0290af57a8f6c7367210ba12d91ae5bd491894e3ec67d5e17841776d\": container with ID starting with 84def4eb0290af57a8f6c7367210ba12d91ae5bd491894e3ec67d5e17841776d not found: ID does not exist" Jul 20 18:22:34.508514 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.508431 2583 scope.go:117] "RemoveContainer" containerID="e3f7276f3a449e39d278bda7e145f42cfca0ee615e1d0c4514a501ac42fa492e" Jul 20 18:22:34.508955 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:22:34.508883 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f7276f3a449e39d278bda7e145f42cfca0ee615e1d0c4514a501ac42fa492e\": container with ID starting with e3f7276f3a449e39d278bda7e145f42cfca0ee615e1d0c4514a501ac42fa492e not found: ID does not exist" containerID="e3f7276f3a449e39d278bda7e145f42cfca0ee615e1d0c4514a501ac42fa492e" Jul 20 18:22:34.508955 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.508919 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f7276f3a449e39d278bda7e145f42cfca0ee615e1d0c4514a501ac42fa492e"} err="failed to get container status \"e3f7276f3a449e39d278bda7e145f42cfca0ee615e1d0c4514a501ac42fa492e\": rpc error: code = NotFound desc = could not find container \"e3f7276f3a449e39d278bda7e145f42cfca0ee615e1d0c4514a501ac42fa492e\": container with ID starting with e3f7276f3a449e39d278bda7e145f42cfca0ee615e1d0c4514a501ac42fa492e not found: ID does not exist" Jul 20 18:22:34.511404 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.511385 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc"] Jul 20 18:22:34.514536 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:34.514516 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bd66d6f4-openshift-default-9449d948c-5k9tc"] Jul 20 18:22:36.170352 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:36.170319 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" path="/var/lib/kubelet/pods/e32c1bca-620e-42ab-a9bc-d7d6788c4f9f/volumes" Jul 20 18:22:36.843547 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:36.843512 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:22:45.571884 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:45.571849 2583 generic.go:356] "Generic (PLEG): container finished" podID="15d297c0-bdaa-4d74-9eed-6c4f922aeb62" containerID="faece67d3baf471ef06a91c563805d04d36503fa27829103a03f0582515ac734" exitCode=0 Jul 20 18:22:45.572308 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:45.571906 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" event={"ID":"15d297c0-bdaa-4d74-9eed-6c4f922aeb62","Type":"ContainerDied","Data":"faece67d3baf471ef06a91c563805d04d36503fa27829103a03f0582515ac734"} Jul 20 18:22:45.572308 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:45.572297 2583 scope.go:117] "RemoveContainer" containerID="faece67d3baf471ef06a91c563805d04d36503fa27829103a03f0582515ac734" Jul 20 18:22:45.573406 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:45.573376 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:22:45.573505 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:45.573486 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:22:45.573547 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:45.573534 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:22:46.578946 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:46.578907 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" event={"ID":"15d297c0-bdaa-4d74-9eed-6c4f922aeb62","Type":"ContainerStarted","Data":"45d5d5890ed9cf1c56258b337acbde9b9ef987b43afb5e20c1619585d72dfbff"} Jul 20 18:22:46.824260 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:46.824223 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:46.825430 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:46.825402 2583 patch_prober.go:28] interesting pod/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 20 18:22:46.825642 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:46.825615 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" podUID="15d297c0-bdaa-4d74-9eed-6c4f922aeb62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 20 18:22:47.479856 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:47.479817 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:47.824533 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:47.824434 2583 patch_prober.go:28] interesting pod/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 20 18:22:47.825124 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:47.825090 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" podUID="15d297c0-bdaa-4d74-9eed-6c4f922aeb62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 20 18:22:48.824367 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:48.824323 2583 patch_prober.go:28] interesting pod/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 20 18:22:48.824654 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:48.824626 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" podUID="15d297c0-bdaa-4d74-9eed-6c4f922aeb62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 20 18:22:49.824533 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:49.824485 2583 patch_prober.go:28] interesting pod/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 20 18:22:49.824764 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:49.824556 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" podUID="15d297c0-bdaa-4d74-9eed-6c4f922aeb62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 20 18:22:50.827566 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:50.827534 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:22:50.828526 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:22:50.828508 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:23:10.445357 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:10.445320 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:23:11.942825 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:11.942793 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd"] Jul 20 18:23:11.943200 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:11.942994 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" podUID="15d297c0-bdaa-4d74-9eed-6c4f922aeb62" containerName="istio-proxy" containerID="cri-o://45d5d5890ed9cf1c56258b337acbde9b9ef987b43afb5e20c1619585d72dfbff" gracePeriod=30 Jul 20 18:23:12.165485 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:12.165455 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 18:23:12.175538 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:12.175508 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 18:23:17.194533 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.194505 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:23:17.351418 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.351322 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-data\") pod \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " Jul 20 18:23:17.351418 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.351363 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istiod-ca-cert\") pod \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " Jul 20 18:23:17.351418 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.351403 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-credential-socket\") pod \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " Jul 20 18:23:17.351698 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.351448 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-workload-certs\") pod \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " Jul 20 18:23:17.351698 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.351466 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-envoy\") pod \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " Jul 20 18:23:17.351698 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.351484 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-token\") pod \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " Jul 20 18:23:17.351698 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.351504 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brkwt\" (UniqueName: \"kubernetes.io/projected/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-kube-api-access-brkwt\") pod \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " Jul 20 18:23:17.351698 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.351523 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-workload-socket\") pod \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " Jul 20 18:23:17.351698 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.351565 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-podinfo\") pod \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\" (UID: \"15d297c0-bdaa-4d74-9eed-6c4f922aeb62\") " Jul 20 18:23:17.352044 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.351831 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "15d297c0-bdaa-4d74-9eed-6c4f922aeb62" (UID: "15d297c0-bdaa-4d74-9eed-6c4f922aeb62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:23:17.352044 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.351890 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "15d297c0-bdaa-4d74-9eed-6c4f922aeb62" (UID: "15d297c0-bdaa-4d74-9eed-6c4f922aeb62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 18:23:17.352156 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.352122 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "15d297c0-bdaa-4d74-9eed-6c4f922aeb62" (UID: "15d297c0-bdaa-4d74-9eed-6c4f922aeb62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:23:17.352222 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.352172 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "15d297c0-bdaa-4d74-9eed-6c4f922aeb62" (UID: "15d297c0-bdaa-4d74-9eed-6c4f922aeb62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:23:17.352522 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.352312 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-data" (OuterVolumeSpecName: "istio-data") pod "15d297c0-bdaa-4d74-9eed-6c4f922aeb62" (UID: "15d297c0-bdaa-4d74-9eed-6c4f922aeb62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:23:17.352664 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.352539 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-credential-socket\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:23:17.352664 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.352559 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-workload-certs\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:23:17.352664 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.352573 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-workload-socket\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:23:17.352664 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.352588 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-data\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:23:17.352664 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.352602 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istiod-ca-cert\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:23:17.353982 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.353945 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-token" (OuterVolumeSpecName: "istio-token") pod "15d297c0-bdaa-4d74-9eed-6c4f922aeb62" (UID: "15d297c0-bdaa-4d74-9eed-6c4f922aeb62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:23:17.354088 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.354017 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-kube-api-access-brkwt" (OuterVolumeSpecName: "kube-api-access-brkwt") pod "15d297c0-bdaa-4d74-9eed-6c4f922aeb62" (UID: "15d297c0-bdaa-4d74-9eed-6c4f922aeb62"). InnerVolumeSpecName "kube-api-access-brkwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:23:17.354181 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.354163 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "15d297c0-bdaa-4d74-9eed-6c4f922aeb62" (UID: "15d297c0-bdaa-4d74-9eed-6c4f922aeb62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:23:17.354310 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.354295 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "15d297c0-bdaa-4d74-9eed-6c4f922aeb62" (UID: "15d297c0-bdaa-4d74-9eed-6c4f922aeb62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 18:23:17.453167 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.453137 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-envoy\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:23:17.453167 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.453160 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-token\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:23:17.453167 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.453170 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brkwt\" (UniqueName: \"kubernetes.io/projected/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-kube-api-access-brkwt\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:23:17.453397 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.453179 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d297c0-bdaa-4d74-9eed-6c4f922aeb62-istio-podinfo\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:23:17.729708 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.729671 2583 generic.go:356] "Generic (PLEG): container finished" podID="15d297c0-bdaa-4d74-9eed-6c4f922aeb62" containerID="45d5d5890ed9cf1c56258b337acbde9b9ef987b43afb5e20c1619585d72dfbff" exitCode=0 Jul 20 18:23:17.729899 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.729721 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" Jul 20 18:23:17.729899 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.729722 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" event={"ID":"15d297c0-bdaa-4d74-9eed-6c4f922aeb62","Type":"ContainerDied","Data":"45d5d5890ed9cf1c56258b337acbde9b9ef987b43afb5e20c1619585d72dfbff"} Jul 20 18:23:17.729899 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.729791 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd" event={"ID":"15d297c0-bdaa-4d74-9eed-6c4f922aeb62","Type":"ContainerDied","Data":"ed82a912008e36555a08f83704bb8b01952ba5b11e3fcf8b544ed21ba60874aa"} Jul 20 18:23:17.729899 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.729815 2583 scope.go:117] "RemoveContainer" containerID="45d5d5890ed9cf1c56258b337acbde9b9ef987b43afb5e20c1619585d72dfbff" Jul 20 18:23:17.739052 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.739033 2583 scope.go:117] "RemoveContainer" containerID="faece67d3baf471ef06a91c563805d04d36503fa27829103a03f0582515ac734" Jul 20 18:23:17.746845 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.746829 2583 scope.go:117] "RemoveContainer" containerID="45d5d5890ed9cf1c56258b337acbde9b9ef987b43afb5e20c1619585d72dfbff" Jul 20 18:23:17.747060 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:23:17.747043 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d5d5890ed9cf1c56258b337acbde9b9ef987b43afb5e20c1619585d72dfbff\": container with ID starting with 45d5d5890ed9cf1c56258b337acbde9b9ef987b43afb5e20c1619585d72dfbff not found: ID does not exist" containerID="45d5d5890ed9cf1c56258b337acbde9b9ef987b43afb5e20c1619585d72dfbff" Jul 20 18:23:17.747104 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.747067 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d5d5890ed9cf1c56258b337acbde9b9ef987b43afb5e20c1619585d72dfbff"} err="failed to get container status \"45d5d5890ed9cf1c56258b337acbde9b9ef987b43afb5e20c1619585d72dfbff\": rpc error: code = NotFound desc = could not find container \"45d5d5890ed9cf1c56258b337acbde9b9ef987b43afb5e20c1619585d72dfbff\": container with ID starting with 45d5d5890ed9cf1c56258b337acbde9b9ef987b43afb5e20c1619585d72dfbff not found: ID does not exist" Jul 20 18:23:17.747104 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.747084 2583 scope.go:117] "RemoveContainer" containerID="faece67d3baf471ef06a91c563805d04d36503fa27829103a03f0582515ac734" Jul 20 18:23:17.747355 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:23:17.747335 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faece67d3baf471ef06a91c563805d04d36503fa27829103a03f0582515ac734\": container with ID starting with faece67d3baf471ef06a91c563805d04d36503fa27829103a03f0582515ac734 not found: ID does not exist" containerID="faece67d3baf471ef06a91c563805d04d36503fa27829103a03f0582515ac734" Jul 20 18:23:17.747406 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.747361 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faece67d3baf471ef06a91c563805d04d36503fa27829103a03f0582515ac734"} err="failed to get container status \"faece67d3baf471ef06a91c563805d04d36503fa27829103a03f0582515ac734\": rpc error: code = NotFound desc = could not find container \"faece67d3baf471ef06a91c563805d04d36503fa27829103a03f0582515ac734\": container with ID starting with faece67d3baf471ef06a91c563805d04d36503fa27829103a03f0582515ac734 not found: ID does not exist" Jul 20 18:23:17.752547 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.752525 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd"] Jul 20 18:23:17.762009 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:17.761989 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e0a9248b-openshift-default-86d4c7685f-pcdfd"] Jul 20 18:23:18.170213 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:23:18.170135 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15d297c0-bdaa-4d74-9eed-6c4f922aeb62" path="/var/lib/kubelet/pods/15d297c0-bdaa-4d74-9eed-6c4f922aeb62/volumes" Jul 20 18:24:37.055437 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:37.055404 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:24:46.251660 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:46.251625 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:24:50.172437 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.172407 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk"] Jul 20 18:24:50.172831 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.172709 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15d297c0-bdaa-4d74-9eed-6c4f922aeb62" containerName="istio-proxy" Jul 20 18:24:50.172831 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.172722 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="15d297c0-bdaa-4d74-9eed-6c4f922aeb62" containerName="istio-proxy" Jul 20 18:24:50.172831 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.172756 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" containerName="istio-proxy" Jul 20 18:24:50.172831 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.172764 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" containerName="istio-proxy" Jul 20 18:24:50.172831 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.172779 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" containerName="istio-proxy" Jul 20 18:24:50.172831 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.172785 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" containerName="istio-proxy" Jul 20 18:24:50.172831 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.172792 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15d297c0-bdaa-4d74-9eed-6c4f922aeb62" containerName="istio-proxy" Jul 20 18:24:50.172831 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.172798 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="15d297c0-bdaa-4d74-9eed-6c4f922aeb62" containerName="istio-proxy" Jul 20 18:24:50.173073 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.172847 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" containerName="istio-proxy" Jul 20 18:24:50.173073 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.172856 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="15d297c0-bdaa-4d74-9eed-6c4f922aeb62" containerName="istio-proxy" Jul 20 18:24:50.175530 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.175515 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.178222 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.178200 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fca5d313-openshift-default-dockercfg-zxtc8\"" Jul 20 18:24:50.196408 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.196383 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk"] Jul 20 18:24:50.319372 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.319328 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-credential-socket\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.319372 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.319374 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-workload-socket\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.319609 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.319402 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-token\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.319609 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.319453 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tklgc\" (UniqueName: \"kubernetes.io/projected/5526fd90-06aa-4f2f-803b-868064c36bdd-kube-api-access-tklgc\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.319609 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.319494 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-workload-certs\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.319609 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.319519 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-envoy\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.319609 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.319540 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5526fd90-06aa-4f2f-803b-868064c36bdd-istiod-ca-cert\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.319609 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.319566 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-data\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.320104 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.319616 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-podinfo\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.420983 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.420946 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tklgc\" (UniqueName: \"kubernetes.io/projected/5526fd90-06aa-4f2f-803b-868064c36bdd-kube-api-access-tklgc\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.421164 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.420993 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-workload-certs\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.421164 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.421022 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-envoy\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.421164 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.421043 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5526fd90-06aa-4f2f-803b-868064c36bdd-istiod-ca-cert\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.421164 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.421066 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-data\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.421164 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.421108 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-podinfo\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.421164 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.421153 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-credential-socket\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.421515 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.421179 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-workload-socket\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.421515 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.421207 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-token\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.421515 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.421438 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-workload-certs\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.421515 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.421485 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-data\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.421715 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.421617 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-credential-socket\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.421715 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.421679 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-workload-socket\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.421939 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.421918 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5526fd90-06aa-4f2f-803b-868064c36bdd-istiod-ca-cert\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.423339 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.423285 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-envoy\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.423516 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.423497 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-podinfo\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.429547 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.429518 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-token\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.429660 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.429645 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tklgc\" (UniqueName: \"kubernetes.io/projected/5526fd90-06aa-4f2f-803b-868064c36bdd-kube-api-access-tklgc\") pod \"e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.489576 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.489546 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:50.630720 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.630698 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk"] Jul 20 18:24:50.632133 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:24:50.632109 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5526fd90_06aa_4f2f_803b_868064c36bdd.slice/crio-c434ef7327e7f2c6bd3c94f33f750cc2a05839ce4ed3e4aa34219e0daa2cdf25 WatchSource:0}: Error finding container c434ef7327e7f2c6bd3c94f33f750cc2a05839ce4ed3e4aa34219e0daa2cdf25: Status 404 returned error can't find the container with id c434ef7327e7f2c6bd3c94f33f750cc2a05839ce4ed3e4aa34219e0daa2cdf25 Jul 20 18:24:50.634478 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.634433 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:24:50.634581 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.634514 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:24:50.634581 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:50.634552 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:24:51.200355 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:51.200317 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" event={"ID":"5526fd90-06aa-4f2f-803b-868064c36bdd","Type":"ContainerStarted","Data":"c0430dc795001fb2c5ddb6f9d91f94af1ea50e4d9dc8ee3673499af9857c3e9d"} Jul 20 18:24:51.200894 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:51.200374 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" event={"ID":"5526fd90-06aa-4f2f-803b-868064c36bdd","Type":"ContainerStarted","Data":"c434ef7327e7f2c6bd3c94f33f750cc2a05839ce4ed3e4aa34219e0daa2cdf25"} Jul 20 18:24:51.223439 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:51.223394 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" podStartSLOduration=1.223379261 podStartE2EDuration="1.223379261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:24:51.222242025 +0000 UTC m=+2199.594030598" watchObservedRunningTime="2026-07-20 18:24:51.223379261 +0000 UTC m=+2199.595167836" Jul 20 18:24:51.490394 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:51.490304 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:51.495271 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:51.495242 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:52.205797 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:52.205760 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:52.206647 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:52.206629 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:24:58.853843 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:24:58.853808 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:25:07.207590 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:07.207549 2583 patch_prober.go:28] interesting pod/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 20 18:25:07.208130 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:07.207630 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" podUID="5526fd90-06aa-4f2f-803b-868064c36bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 20 18:25:07.290201 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:07.290164 2583 generic.go:356] "Generic (PLEG): container finished" podID="5526fd90-06aa-4f2f-803b-868064c36bdd" containerID="c0430dc795001fb2c5ddb6f9d91f94af1ea50e4d9dc8ee3673499af9857c3e9d" exitCode=0 Jul 20 18:25:07.290397 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:07.290237 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" event={"ID":"5526fd90-06aa-4f2f-803b-868064c36bdd","Type":"ContainerDied","Data":"c0430dc795001fb2c5ddb6f9d91f94af1ea50e4d9dc8ee3673499af9857c3e9d"} Jul 20 18:25:07.290664 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:07.290646 2583 scope.go:117] "RemoveContainer" containerID="c0430dc795001fb2c5ddb6f9d91f94af1ea50e4d9dc8ee3673499af9857c3e9d" Jul 20 18:25:07.291671 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:07.291644 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:25:07.291852 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:07.291719 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:25:07.291852 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:07.291780 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:25:08.296701 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:08.296667 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" event={"ID":"5526fd90-06aa-4f2f-803b-868064c36bdd","Type":"ContainerStarted","Data":"cc3b19b969deac75f9cc675cc7fe57a469d9f928da85a873c45388478211a001"} Jul 20 18:25:08.490192 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:08.490158 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:25:08.490362 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:08.490257 2583 patch_prober.go:28] interesting pod/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 20 18:25:08.490362 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:08.490306 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" podUID="5526fd90-06aa-4f2f-803b-868064c36bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 20 18:25:09.490087 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:09.490048 2583 patch_prober.go:28] interesting pod/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 20 18:25:09.490495 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:09.490103 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" podUID="5526fd90-06aa-4f2f-803b-868064c36bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 20 18:25:10.489912 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:10.489879 2583 patch_prober.go:28] interesting pod/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 20 18:25:10.490085 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:10.489930 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" podUID="5526fd90-06aa-4f2f-803b-868064c36bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 20 18:25:11.490653 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:11.490615 2583 patch_prober.go:28] interesting pod/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 20 18:25:11.491075 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:11.490672 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" podUID="5526fd90-06aa-4f2f-803b-868064c36bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 20 18:25:12.490124 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:12.490084 2583 patch_prober.go:28] interesting pod/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 20 18:25:12.490311 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:12.490154 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" podUID="5526fd90-06aa-4f2f-803b-868064c36bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 20 18:25:13.493939 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:13.493903 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:25:13.494373 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:13.494281 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:25:13.494853 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:13.494835 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:25:18.352947 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:18.352912 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:25:26.920915 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:26.920882 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk"] Jul 20 18:25:26.921401 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:26.921116 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" podUID="5526fd90-06aa-4f2f-803b-868064c36bdd" containerName="istio-proxy" containerID="cri-o://cc3b19b969deac75f9cc675cc7fe57a469d9f928da85a873c45388478211a001" gracePeriod=30 Jul 20 18:25:28.497060 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:28.497029 2583 patch_prober.go:28] interesting pod/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 20 18:25:28.497441 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:28.497082 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" podUID="5526fd90-06aa-4f2f-803b-868064c36bdd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 18:25:32.163091 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.163062 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:25:32.268907 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.268827 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5526fd90-06aa-4f2f-803b-868064c36bdd-istiod-ca-cert\") pod \"5526fd90-06aa-4f2f-803b-868064c36bdd\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " Jul 20 18:25:32.268907 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.268862 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-podinfo\") pod \"5526fd90-06aa-4f2f-803b-868064c36bdd\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " Jul 20 18:25:32.268907 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.268896 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-data\") pod \"5526fd90-06aa-4f2f-803b-868064c36bdd\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " Jul 20 18:25:32.269177 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.268914 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-credential-socket\") pod \"5526fd90-06aa-4f2f-803b-868064c36bdd\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " Jul 20 18:25:32.269177 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.268938 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-workload-socket\") pod \"5526fd90-06aa-4f2f-803b-868064c36bdd\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " Jul 20 18:25:32.269177 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.268963 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-envoy\") pod \"5526fd90-06aa-4f2f-803b-868064c36bdd\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " Jul 20 18:25:32.269177 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.269069 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-token\") pod \"5526fd90-06aa-4f2f-803b-868064c36bdd\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " Jul 20 18:25:32.269177 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.269125 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tklgc\" (UniqueName: \"kubernetes.io/projected/5526fd90-06aa-4f2f-803b-868064c36bdd-kube-api-access-tklgc\") pod \"5526fd90-06aa-4f2f-803b-868064c36bdd\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " Jul 20 18:25:32.269177 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.269154 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-workload-certs\") pod \"5526fd90-06aa-4f2f-803b-868064c36bdd\" (UID: \"5526fd90-06aa-4f2f-803b-868064c36bdd\") " Jul 20 18:25:32.269506 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.269226 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5526fd90-06aa-4f2f-803b-868064c36bdd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5526fd90-06aa-4f2f-803b-868064c36bdd" (UID: "5526fd90-06aa-4f2f-803b-868064c36bdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 18:25:32.269506 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.269306 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5526fd90-06aa-4f2f-803b-868064c36bdd" (UID: "5526fd90-06aa-4f2f-803b-868064c36bdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:25:32.269506 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.269317 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5526fd90-06aa-4f2f-803b-868064c36bdd" (UID: "5526fd90-06aa-4f2f-803b-868064c36bdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:25:32.269506 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.269425 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5526fd90-06aa-4f2f-803b-868064c36bdd" (UID: "5526fd90-06aa-4f2f-803b-868064c36bdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:25:32.269506 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.269443 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-credential-socket\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:25:32.269506 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.269462 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-workload-socket\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:25:32.269506 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.269477 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5526fd90-06aa-4f2f-803b-868064c36bdd-istiod-ca-cert\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:25:32.269920 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.269771 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-data" (OuterVolumeSpecName: "istio-data") pod "5526fd90-06aa-4f2f-803b-868064c36bdd" (UID: "5526fd90-06aa-4f2f-803b-868064c36bdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:25:32.271151 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.271123 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5526fd90-06aa-4f2f-803b-868064c36bdd" (UID: "5526fd90-06aa-4f2f-803b-868064c36bdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 18:25:32.271311 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.271290 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-token" (OuterVolumeSpecName: "istio-token") pod "5526fd90-06aa-4f2f-803b-868064c36bdd" (UID: "5526fd90-06aa-4f2f-803b-868064c36bdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:25:32.271383 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.271314 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5526fd90-06aa-4f2f-803b-868064c36bdd" (UID: "5526fd90-06aa-4f2f-803b-868064c36bdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:25:32.271503 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.271484 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5526fd90-06aa-4f2f-803b-868064c36bdd-kube-api-access-tklgc" (OuterVolumeSpecName: "kube-api-access-tklgc") pod "5526fd90-06aa-4f2f-803b-868064c36bdd" (UID: "5526fd90-06aa-4f2f-803b-868064c36bdd"). InnerVolumeSpecName "kube-api-access-tklgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:25:32.370222 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.370189 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-token\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:25:32.370222 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.370218 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tklgc\" (UniqueName: \"kubernetes.io/projected/5526fd90-06aa-4f2f-803b-868064c36bdd-kube-api-access-tklgc\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:25:32.370222 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.370228 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-workload-certs\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:25:32.370450 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.370238 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-podinfo\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:25:32.370450 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.370245 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-data\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:25:32.370450 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.370255 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5526fd90-06aa-4f2f-803b-868064c36bdd-istio-envoy\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:25:32.427813 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.427777 2583 generic.go:356] "Generic (PLEG): container finished" podID="5526fd90-06aa-4f2f-803b-868064c36bdd" containerID="cc3b19b969deac75f9cc675cc7fe57a469d9f928da85a873c45388478211a001" exitCode=0 Jul 20 18:25:32.427956 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.427830 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" event={"ID":"5526fd90-06aa-4f2f-803b-868064c36bdd","Type":"ContainerDied","Data":"cc3b19b969deac75f9cc675cc7fe57a469d9f928da85a873c45388478211a001"} Jul 20 18:25:32.427956 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.427875 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" event={"ID":"5526fd90-06aa-4f2f-803b-868064c36bdd","Type":"ContainerDied","Data":"c434ef7327e7f2c6bd3c94f33f750cc2a05839ce4ed3e4aa34219e0daa2cdf25"} Jul 20 18:25:32.427956 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.427899 2583 scope.go:117] "RemoveContainer" containerID="cc3b19b969deac75f9cc675cc7fe57a469d9f928da85a873c45388478211a001" Jul 20 18:25:32.427956 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.427833 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk" Jul 20 18:25:32.438122 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.438046 2583 scope.go:117] "RemoveContainer" containerID="c0430dc795001fb2c5ddb6f9d91f94af1ea50e4d9dc8ee3673499af9857c3e9d" Jul 20 18:25:32.446939 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.446919 2583 scope.go:117] "RemoveContainer" containerID="cc3b19b969deac75f9cc675cc7fe57a469d9f928da85a873c45388478211a001" Jul 20 18:25:32.447183 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:25:32.447164 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3b19b969deac75f9cc675cc7fe57a469d9f928da85a873c45388478211a001\": container with ID starting with cc3b19b969deac75f9cc675cc7fe57a469d9f928da85a873c45388478211a001 not found: ID does not exist" containerID="cc3b19b969deac75f9cc675cc7fe57a469d9f928da85a873c45388478211a001" Jul 20 18:25:32.447226 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.447192 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3b19b969deac75f9cc675cc7fe57a469d9f928da85a873c45388478211a001"} err="failed to get container status \"cc3b19b969deac75f9cc675cc7fe57a469d9f928da85a873c45388478211a001\": rpc error: code = NotFound desc = could not find container \"cc3b19b969deac75f9cc675cc7fe57a469d9f928da85a873c45388478211a001\": container with ID starting with cc3b19b969deac75f9cc675cc7fe57a469d9f928da85a873c45388478211a001 not found: ID does not exist" Jul 20 18:25:32.447226 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.447209 2583 scope.go:117] "RemoveContainer" containerID="c0430dc795001fb2c5ddb6f9d91f94af1ea50e4d9dc8ee3673499af9857c3e9d" Jul 20 18:25:32.447433 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:25:32.447416 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0430dc795001fb2c5ddb6f9d91f94af1ea50e4d9dc8ee3673499af9857c3e9d\": container with ID starting with c0430dc795001fb2c5ddb6f9d91f94af1ea50e4d9dc8ee3673499af9857c3e9d not found: ID does not exist" containerID="c0430dc795001fb2c5ddb6f9d91f94af1ea50e4d9dc8ee3673499af9857c3e9d" Jul 20 18:25:32.447470 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.447441 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0430dc795001fb2c5ddb6f9d91f94af1ea50e4d9dc8ee3673499af9857c3e9d"} err="failed to get container status \"c0430dc795001fb2c5ddb6f9d91f94af1ea50e4d9dc8ee3673499af9857c3e9d\": rpc error: code = NotFound desc = could not find container \"c0430dc795001fb2c5ddb6f9d91f94af1ea50e4d9dc8ee3673499af9857c3e9d\": container with ID starting with c0430dc795001fb2c5ddb6f9d91f94af1ea50e4d9dc8ee3673499af9857c3e9d not found: ID does not exist" Jul 20 18:25:32.454655 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.454604 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk"] Jul 20 18:25:32.460446 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:32.460427 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fca5d313-openshift-default-cf67c9bd7-vxsvk"] Jul 20 18:25:34.170603 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:34.170573 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5526fd90-06aa-4f2f-803b-868064c36bdd" path="/var/lib/kubelet/pods/5526fd90-06aa-4f2f-803b-868064c36bdd/volumes" Jul 20 18:25:46.785367 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.785334 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5"] Jul 20 18:25:46.785865 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.785650 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5526fd90-06aa-4f2f-803b-868064c36bdd" containerName="istio-proxy" Jul 20 18:25:46.785865 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.785662 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5526fd90-06aa-4f2f-803b-868064c36bdd" containerName="istio-proxy" Jul 20 18:25:46.785865 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.785676 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5526fd90-06aa-4f2f-803b-868064c36bdd" containerName="istio-proxy" Jul 20 18:25:46.785865 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.785685 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5526fd90-06aa-4f2f-803b-868064c36bdd" containerName="istio-proxy" Jul 20 18:25:46.785865 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.785786 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5526fd90-06aa-4f2f-803b-868064c36bdd" containerName="istio-proxy" Jul 20 18:25:46.785865 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.785797 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e32c1bca-620e-42ab-a9bc-d7d6788c4f9f" containerName="istio-proxy" Jul 20 18:25:46.785865 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.785805 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="15d297c0-bdaa-4d74-9eed-6c4f922aeb62" containerName="istio-proxy" Jul 20 18:25:46.785865 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.785813 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5526fd90-06aa-4f2f-803b-868064c36bdd" containerName="istio-proxy" Jul 20 18:25:46.788598 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.788569 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.791645 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.791625 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2be07642-openshift-default-dockercfg-s6vgj\"" Jul 20 18:25:46.806630 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.806607 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5"] Jul 20 18:25:46.885065 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.885027 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-credential-socket\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.885231 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.885074 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-envoy\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.885231 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.885097 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-workload-socket\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.885231 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.885178 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-data\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.885231 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.885219 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl9xx\" (UniqueName: \"kubernetes.io/projected/ec3dee2a-a636-43e8-97e0-e29003ea169a-kube-api-access-dl9xx\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.885374 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.885267 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-token\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.885374 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.885288 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-workload-certs\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.885374 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.885319 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-podinfo\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.885374 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.885339 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3dee2a-a636-43e8-97e0-e29003ea169a-istiod-ca-cert\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.986430 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.986388 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-credential-socket\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.986430 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.986431 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-envoy\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.986694 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.986455 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-workload-socket\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.986694 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.986489 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-data\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.986694 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.986522 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl9xx\" (UniqueName: \"kubernetes.io/projected/ec3dee2a-a636-43e8-97e0-e29003ea169a-kube-api-access-dl9xx\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.986694 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.986565 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-token\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.986694 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.986620 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-workload-certs\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.987042 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.986782 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-podinfo\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.987042 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.986825 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3dee2a-a636-43e8-97e0-e29003ea169a-istiod-ca-cert\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.987042 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.986834 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-credential-socket\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.987042 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.986886 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-data\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.987042 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.986990 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-workload-certs\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.987257 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.987234 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-workload-socket\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.987474 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.987457 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3dee2a-a636-43e8-97e0-e29003ea169a-istiod-ca-cert\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.988768 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.988746 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-envoy\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.989019 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.989001 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-podinfo\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.995176 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.995154 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-token\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:46.995483 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:46.995446 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl9xx\" (UniqueName: \"kubernetes.io/projected/ec3dee2a-a636-43e8-97e0-e29003ea169a-kube-api-access-dl9xx\") pod \"e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:47.098121 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:47.098031 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:47.261371 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:47.261338 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5"] Jul 20 18:25:47.263495 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:25:47.263465 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3dee2a_a636_43e8_97e0_e29003ea169a.slice/crio-f4f55cdec377b99067681139a49b7566af0f2f7d20486cfd9ac365b1caaeefe5 WatchSource:0}: Error finding container f4f55cdec377b99067681139a49b7566af0f2f7d20486cfd9ac365b1caaeefe5: Status 404 returned error can't find the container with id f4f55cdec377b99067681139a49b7566af0f2f7d20486cfd9ac365b1caaeefe5 Jul 20 18:25:47.265993 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:47.265958 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:25:47.266084 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:47.266021 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:25:47.266084 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:47.266048 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:25:47.505716 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:47.505681 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" event={"ID":"ec3dee2a-a636-43e8-97e0-e29003ea169a","Type":"ContainerStarted","Data":"c0e686e0f31b1affb4ccebd4945ed0a3aebcdbc8618556e58140e61533a33a6e"} Jul 20 18:25:47.505901 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:47.505759 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" event={"ID":"ec3dee2a-a636-43e8-97e0-e29003ea169a","Type":"ContainerStarted","Data":"f4f55cdec377b99067681139a49b7566af0f2f7d20486cfd9ac365b1caaeefe5"} Jul 20 18:25:47.529486 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:47.529436 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" podStartSLOduration=1.5294219500000001 podStartE2EDuration="1.52942195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:25:47.526973072 +0000 UTC m=+2255.898761644" watchObservedRunningTime="2026-07-20 18:25:47.52942195 +0000 UTC m=+2255.901210523" Jul 20 18:25:48.098905 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:48.098869 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:48.103630 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:48.103607 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:48.510628 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:48.510598 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:48.511574 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:48.511556 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:25:54.952544 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:25:54.952507 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:26:02.579790 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:02.579749 2583 generic.go:356] "Generic (PLEG): container finished" podID="ec3dee2a-a636-43e8-97e0-e29003ea169a" containerID="c0e686e0f31b1affb4ccebd4945ed0a3aebcdbc8618556e58140e61533a33a6e" exitCode=0 Jul 20 18:26:02.580181 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:02.579822 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" event={"ID":"ec3dee2a-a636-43e8-97e0-e29003ea169a","Type":"ContainerDied","Data":"c0e686e0f31b1affb4ccebd4945ed0a3aebcdbc8618556e58140e61533a33a6e"} Jul 20 18:26:02.580230 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:02.580199 2583 scope.go:117] "RemoveContainer" containerID="c0e686e0f31b1affb4ccebd4945ed0a3aebcdbc8618556e58140e61533a33a6e" Jul 20 18:26:02.581191 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:02.581167 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:26:02.581243 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:02.581228 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:26:02.581279 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:02.581261 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:26:03.510710 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:03.510671 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:26:03.586210 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:03.586175 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" event={"ID":"ec3dee2a-a636-43e8-97e0-e29003ea169a","Type":"ContainerStarted","Data":"573efa10129fa9659dd0fec4b95668fcf99e868d4f0527794d96c425c4809fee"} Jul 20 18:26:04.098604 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:04.098572 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:26:04.099843 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:04.099817 2583 patch_prober.go:28] interesting pod/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 20 18:26:04.099953 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:04.099863 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" podUID="ec3dee2a-a636-43e8-97e0-e29003ea169a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 20 18:26:05.099103 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:05.099068 2583 patch_prober.go:28] interesting pod/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 20 18:26:05.099683 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:05.099653 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" podUID="ec3dee2a-a636-43e8-97e0-e29003ea169a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 20 18:26:06.098576 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:06.098541 2583 patch_prober.go:28] interesting pod/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 20 18:26:06.098797 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:06.098596 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" podUID="ec3dee2a-a636-43e8-97e0-e29003ea169a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 20 18:26:06.852317 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:06.852278 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:26:07.099162 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:07.099131 2583 patch_prober.go:28] interesting pod/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 20 18:26:07.099340 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:07.099199 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" podUID="ec3dee2a-a636-43e8-97e0-e29003ea169a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 20 18:26:08.102420 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:08.102385 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:26:08.102915 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:08.102760 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:26:08.103347 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:08.103328 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:26:14.149642 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:14.149610 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:26:31.074912 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:31.074869 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:26:41.781996 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:26:41.781959 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:27:34.689858 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:34.689825 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5"] Jul 20 18:27:34.690380 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:34.690041 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" podUID="ec3dee2a-a636-43e8-97e0-e29003ea169a" containerName="istio-proxy" containerID="cri-o://573efa10129fa9659dd0fec4b95668fcf99e868d4f0527794d96c425c4809fee" gracePeriod=30 Jul 20 18:27:38.105159 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:38.105122 2583 patch_prober.go:28] interesting pod/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 20 18:27:38.105636 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:38.105198 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" podUID="ec3dee2a-a636-43e8-97e0-e29003ea169a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 18:27:38.450174 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:38.450141 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:27:39.934257 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:39.934232 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:27:40.040897 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.040814 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-workload-socket\") pod \"ec3dee2a-a636-43e8-97e0-e29003ea169a\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " Jul 20 18:27:40.040897 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.040848 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-workload-certs\") pod \"ec3dee2a-a636-43e8-97e0-e29003ea169a\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " Jul 20 18:27:40.040897 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.040876 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl9xx\" (UniqueName: \"kubernetes.io/projected/ec3dee2a-a636-43e8-97e0-e29003ea169a-kube-api-access-dl9xx\") pod \"ec3dee2a-a636-43e8-97e0-e29003ea169a\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " Jul 20 18:27:40.041179 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.040901 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-data\") pod \"ec3dee2a-a636-43e8-97e0-e29003ea169a\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " Jul 20 18:27:40.041179 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.040935 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-credential-socket\") pod \"ec3dee2a-a636-43e8-97e0-e29003ea169a\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " Jul 20 18:27:40.041179 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.040981 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-envoy\") pod \"ec3dee2a-a636-43e8-97e0-e29003ea169a\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " Jul 20 18:27:40.041179 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.041014 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-podinfo\") pod \"ec3dee2a-a636-43e8-97e0-e29003ea169a\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " Jul 20 18:27:40.041179 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.041078 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3dee2a-a636-43e8-97e0-e29003ea169a-istiod-ca-cert\") pod \"ec3dee2a-a636-43e8-97e0-e29003ea169a\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " Jul 20 18:27:40.041179 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.041107 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-token\") pod \"ec3dee2a-a636-43e8-97e0-e29003ea169a\" (UID: \"ec3dee2a-a636-43e8-97e0-e29003ea169a\") " Jul 20 18:27:40.041179 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.041119 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ec3dee2a-a636-43e8-97e0-e29003ea169a" (UID: "ec3dee2a-a636-43e8-97e0-e29003ea169a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:27:40.041537 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.041191 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ec3dee2a-a636-43e8-97e0-e29003ea169a" (UID: "ec3dee2a-a636-43e8-97e0-e29003ea169a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:27:40.041537 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.041256 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ec3dee2a-a636-43e8-97e0-e29003ea169a" (UID: "ec3dee2a-a636-43e8-97e0-e29003ea169a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:27:40.041537 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.041380 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-workload-socket\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:27:40.041537 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.041398 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-workload-certs\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:27:40.041537 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.041411 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-credential-socket\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:27:40.041537 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.041440 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3dee2a-a636-43e8-97e0-e29003ea169a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ec3dee2a-a636-43e8-97e0-e29003ea169a" (UID: "ec3dee2a-a636-43e8-97e0-e29003ea169a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 18:27:40.041826 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.041649 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-data" (OuterVolumeSpecName: "istio-data") pod "ec3dee2a-a636-43e8-97e0-e29003ea169a" (UID: "ec3dee2a-a636-43e8-97e0-e29003ea169a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:27:40.043183 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.043154 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ec3dee2a-a636-43e8-97e0-e29003ea169a" (UID: "ec3dee2a-a636-43e8-97e0-e29003ea169a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:27:40.043311 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.043290 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3dee2a-a636-43e8-97e0-e29003ea169a-kube-api-access-dl9xx" (OuterVolumeSpecName: "kube-api-access-dl9xx") pod "ec3dee2a-a636-43e8-97e0-e29003ea169a" (UID: "ec3dee2a-a636-43e8-97e0-e29003ea169a"). InnerVolumeSpecName "kube-api-access-dl9xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:27:40.043669 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.043644 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-token" (OuterVolumeSpecName: "istio-token") pod "ec3dee2a-a636-43e8-97e0-e29003ea169a" (UID: "ec3dee2a-a636-43e8-97e0-e29003ea169a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:27:40.043797 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.043642 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ec3dee2a-a636-43e8-97e0-e29003ea169a" (UID: "ec3dee2a-a636-43e8-97e0-e29003ea169a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 18:27:40.053195 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.053169 2583 generic.go:356] "Generic (PLEG): container finished" podID="ec3dee2a-a636-43e8-97e0-e29003ea169a" containerID="573efa10129fa9659dd0fec4b95668fcf99e868d4f0527794d96c425c4809fee" exitCode=0 Jul 20 18:27:40.053307 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.053210 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" Jul 20 18:27:40.053307 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.053254 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" event={"ID":"ec3dee2a-a636-43e8-97e0-e29003ea169a","Type":"ContainerDied","Data":"573efa10129fa9659dd0fec4b95668fcf99e868d4f0527794d96c425c4809fee"} Jul 20 18:27:40.053425 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.053315 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5" event={"ID":"ec3dee2a-a636-43e8-97e0-e29003ea169a","Type":"ContainerDied","Data":"f4f55cdec377b99067681139a49b7566af0f2f7d20486cfd9ac365b1caaeefe5"} Jul 20 18:27:40.053425 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.053346 2583 scope.go:117] "RemoveContainer" containerID="573efa10129fa9659dd0fec4b95668fcf99e868d4f0527794d96c425c4809fee" Jul 20 18:27:40.062286 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.062230 2583 scope.go:117] "RemoveContainer" containerID="c0e686e0f31b1affb4ccebd4945ed0a3aebcdbc8618556e58140e61533a33a6e" Jul 20 18:27:40.069887 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.069871 2583 scope.go:117] "RemoveContainer" containerID="573efa10129fa9659dd0fec4b95668fcf99e868d4f0527794d96c425c4809fee" Jul 20 18:27:40.070126 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:27:40.070106 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573efa10129fa9659dd0fec4b95668fcf99e868d4f0527794d96c425c4809fee\": container with ID starting with 573efa10129fa9659dd0fec4b95668fcf99e868d4f0527794d96c425c4809fee not found: ID does not exist" containerID="573efa10129fa9659dd0fec4b95668fcf99e868d4f0527794d96c425c4809fee" Jul 20 18:27:40.070202 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.070140 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573efa10129fa9659dd0fec4b95668fcf99e868d4f0527794d96c425c4809fee"} err="failed to get container status \"573efa10129fa9659dd0fec4b95668fcf99e868d4f0527794d96c425c4809fee\": rpc error: code = NotFound desc = could not find container \"573efa10129fa9659dd0fec4b95668fcf99e868d4f0527794d96c425c4809fee\": container with ID starting with 573efa10129fa9659dd0fec4b95668fcf99e868d4f0527794d96c425c4809fee not found: ID does not exist" Jul 20 18:27:40.070202 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.070165 2583 scope.go:117] "RemoveContainer" containerID="c0e686e0f31b1affb4ccebd4945ed0a3aebcdbc8618556e58140e61533a33a6e" Jul 20 18:27:40.070415 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:27:40.070398 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e686e0f31b1affb4ccebd4945ed0a3aebcdbc8618556e58140e61533a33a6e\": container with ID starting with c0e686e0f31b1affb4ccebd4945ed0a3aebcdbc8618556e58140e61533a33a6e not found: ID does not exist" containerID="c0e686e0f31b1affb4ccebd4945ed0a3aebcdbc8618556e58140e61533a33a6e" Jul 20 18:27:40.070460 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.070420 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e686e0f31b1affb4ccebd4945ed0a3aebcdbc8618556e58140e61533a33a6e"} err="failed to get container status \"c0e686e0f31b1affb4ccebd4945ed0a3aebcdbc8618556e58140e61533a33a6e\": rpc error: code = NotFound desc = could not find container \"c0e686e0f31b1affb4ccebd4945ed0a3aebcdbc8618556e58140e61533a33a6e\": container with ID starting with c0e686e0f31b1affb4ccebd4945ed0a3aebcdbc8618556e58140e61533a33a6e not found: ID does not exist" Jul 20 18:27:40.101132 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.101100 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5"] Jul 20 18:27:40.107252 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.107229 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2be07642-openshift-default-76cf4bd987-6p6l5"] Jul 20 18:27:40.142616 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.142592 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3dee2a-a636-43e8-97e0-e29003ea169a-istiod-ca-cert\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:27:40.142616 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.142611 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-token\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:27:40.142775 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.142621 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl9xx\" (UniqueName: \"kubernetes.io/projected/ec3dee2a-a636-43e8-97e0-e29003ea169a-kube-api-access-dl9xx\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:27:40.142775 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.142630 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-data\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:27:40.142775 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.142639 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-envoy\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:27:40.142775 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.142647 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3dee2a-a636-43e8-97e0-e29003ea169a-istio-podinfo\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:27:40.175833 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:40.175804 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec3dee2a-a636-43e8-97e0-e29003ea169a" path="/var/lib/kubelet/pods/ec3dee2a-a636-43e8-97e0-e29003ea169a/volumes" Jul 20 18:27:48.557672 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:48.557589 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:27:49.099457 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:49.099426 2583 generic.go:356] "Generic (PLEG): container finished" podID="cfb742af-fb9a-4d7f-b199-2d0f3d54fa11" containerID="62c524d7160c610435e6b36ca75b6d7052eec799fb5cbdcb6e9eb34f71ea1017" exitCode=1 Jul 20 18:27:49.099665 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:49.099506 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" event={"ID":"cfb742af-fb9a-4d7f-b199-2d0f3d54fa11","Type":"ContainerDied","Data":"62c524d7160c610435e6b36ca75b6d7052eec799fb5cbdcb6e9eb34f71ea1017"} Jul 20 18:27:49.099933 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:49.099917 2583 scope.go:117] "RemoveContainer" containerID="62c524d7160c610435e6b36ca75b6d7052eec799fb5cbdcb6e9eb34f71ea1017" Jul 20 18:27:50.106573 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:50.106539 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" event={"ID":"cfb742af-fb9a-4d7f-b199-2d0f3d54fa11","Type":"ContainerStarted","Data":"cd9a630ee1761bf95b1596c5c6b953bf6e1f634516a2047a4cc95dd3073c497e"} Jul 20 18:27:50.106976 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:27:50.106753 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" Jul 20 18:28:01.114175 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:01.114141 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7974845875-rvrrz" Jul 20 18:28:01.276962 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:01.276923 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:28:09.180901 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:09.180862 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:28:12.199593 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:12.199569 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 18:28:12.212266 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:12.212245 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 18:28:53.438085 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.438005 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7"] Jul 20 18:28:53.438555 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.438329 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec3dee2a-a636-43e8-97e0-e29003ea169a" containerName="istio-proxy" Jul 20 18:28:53.438555 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.438340 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec3dee2a-a636-43e8-97e0-e29003ea169a" containerName="istio-proxy" Jul 20 18:28:53.438555 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.438355 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec3dee2a-a636-43e8-97e0-e29003ea169a" containerName="istio-proxy" Jul 20 18:28:53.438555 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.438361 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec3dee2a-a636-43e8-97e0-e29003ea169a" containerName="istio-proxy" Jul 20 18:28:53.438555 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.438427 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec3dee2a-a636-43e8-97e0-e29003ea169a" containerName="istio-proxy" Jul 20 18:28:53.441245 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.441227 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.444054 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.444029 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9b1a57-openshift-default-dockercfg-vxv66\"" Jul 20 18:28:53.477120 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.477088 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7"] Jul 20 18:28:53.541786 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.541723 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-data\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.541950 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.541803 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-podinfo\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.541950 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.541838 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-credential-socket\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.541950 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.541865 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-token\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.541950 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.541900 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pq6v\" (UniqueName: \"kubernetes.io/projected/7d753058-9ebc-4f57-89a7-5347c2013fb8-kube-api-access-5pq6v\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.541950 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.541939 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-workload-socket\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.542122 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.541959 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d753058-9ebc-4f57-89a7-5347c2013fb8-istiod-ca-cert\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.542122 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.541989 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-envoy\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.542122 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.542012 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-workload-certs\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.643331 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.643299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-envoy\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.643331 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.643334 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-workload-certs\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.643586 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.643442 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-data\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.643586 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.643509 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-podinfo\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.643586 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.643554 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-credential-socket\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.643586 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.643580 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-token\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.643833 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.643629 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pq6v\" (UniqueName: \"kubernetes.io/projected/7d753058-9ebc-4f57-89a7-5347c2013fb8-kube-api-access-5pq6v\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.643833 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.643701 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-workload-socket\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.643833 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.643763 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d753058-9ebc-4f57-89a7-5347c2013fb8-istiod-ca-cert\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.643833 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.643783 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-workload-certs\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.644039 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.643842 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-data\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.644039 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.643951 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-credential-socket\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.644157 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.644110 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-workload-socket\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.644393 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.644368 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d753058-9ebc-4f57-89a7-5347c2013fb8-istiod-ca-cert\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.645627 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.645609 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-envoy\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.646108 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.646088 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-podinfo\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.659799 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.659772 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pq6v\" (UniqueName: \"kubernetes.io/projected/7d753058-9ebc-4f57-89a7-5347c2013fb8-kube-api-access-5pq6v\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.659918 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.659874 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-token\") pod \"e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.751319 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.751235 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:53.891174 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.891141 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7"] Jul 20 18:28:53.893123 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:28:53.893097 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d753058_9ebc_4f57_89a7_5347c2013fb8.slice/crio-6fc6dfc5c6ba253234f7c007cd83dda5d830e95baff2768e128183a39bdd5188 WatchSource:0}: Error finding container 6fc6dfc5c6ba253234f7c007cd83dda5d830e95baff2768e128183a39bdd5188: Status 404 returned error can't find the container with id 6fc6dfc5c6ba253234f7c007cd83dda5d830e95baff2768e128183a39bdd5188 Jul 20 18:28:53.895248 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.895218 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:28:53.895336 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.895278 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:28:53.895336 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:53.895306 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:28:54.426152 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:54.426112 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" event={"ID":"7d753058-9ebc-4f57-89a7-5347c2013fb8","Type":"ContainerStarted","Data":"699e55ed11fa5bd04b40dc257aaefd48a5daa81557bfc2fcf147ed374ce5d758"} Jul 20 18:28:54.426354 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:54.426164 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" event={"ID":"7d753058-9ebc-4f57-89a7-5347c2013fb8","Type":"ContainerStarted","Data":"6fc6dfc5c6ba253234f7c007cd83dda5d830e95baff2768e128183a39bdd5188"} Jul 20 18:28:54.452840 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:54.452784 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" podStartSLOduration=1.452765257 podStartE2EDuration="1.452765257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:28:54.449856581 +0000 UTC m=+2442.821645154" watchObservedRunningTime="2026-07-20 18:28:54.452765257 +0000 UTC m=+2442.824553831" Jul 20 18:28:54.752361 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:54.752330 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:54.757356 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:54.757329 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:55.431878 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:55.431848 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:28:55.432865 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:28:55.432843 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:29:00.656404 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:00.656368 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:29:08.497249 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:08.497213 2583 generic.go:356] "Generic (PLEG): container finished" podID="7d753058-9ebc-4f57-89a7-5347c2013fb8" containerID="699e55ed11fa5bd04b40dc257aaefd48a5daa81557bfc2fcf147ed374ce5d758" exitCode=0 Jul 20 18:29:08.497677 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:08.497287 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" event={"ID":"7d753058-9ebc-4f57-89a7-5347c2013fb8","Type":"ContainerDied","Data":"699e55ed11fa5bd04b40dc257aaefd48a5daa81557bfc2fcf147ed374ce5d758"} Jul 20 18:29:08.497677 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:08.497673 2583 scope.go:117] "RemoveContainer" containerID="699e55ed11fa5bd04b40dc257aaefd48a5daa81557bfc2fcf147ed374ce5d758" Jul 20 18:29:08.498721 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:08.498689 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:29:08.498826 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:08.498806 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:29:08.498876 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:08.498862 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:29:09.503706 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:09.503667 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" event={"ID":"7d753058-9ebc-4f57-89a7-5347c2013fb8","Type":"ContainerStarted","Data":"6ebb459de3cd9eb50e5cb3dc6b7ed69d7638474d4f01fc020196172e7794b43b"} Jul 20 18:29:09.752073 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:09.752039 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:29:09.753191 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:09.753164 2583 patch_prober.go:28] interesting pod/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 20 18:29:09.753312 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:09.753214 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" podUID="7d753058-9ebc-4f57-89a7-5347c2013fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 20 18:29:10.432674 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:10.432628 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:29:10.752400 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:10.752315 2583 patch_prober.go:28] interesting pod/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 20 18:29:10.752400 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:10.752369 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" podUID="7d753058-9ebc-4f57-89a7-5347c2013fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 20 18:29:11.752071 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:11.752037 2583 patch_prober.go:28] interesting pod/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 20 18:29:11.752249 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:11.752089 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" podUID="7d753058-9ebc-4f57-89a7-5347c2013fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 20 18:29:12.751996 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:12.751964 2583 patch_prober.go:28] interesting pod/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 20 18:29:12.752405 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:12.752019 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" podUID="7d753058-9ebc-4f57-89a7-5347c2013fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 20 18:29:13.755748 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:13.755700 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:29:13.756719 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:13.756699 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:29:14.251564 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:14.251508 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:29:22.498336 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.498292 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5"] Jul 20 18:29:22.498764 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.498701 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec3dee2a-a636-43e8-97e0-e29003ea169a" containerName="istio-proxy" Jul 20 18:29:22.501460 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.501443 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.504164 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.504143 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9b1a57\"/\"openshift-service-ca.crt\"" Jul 20 18:29:22.505047 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.505016 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9b1a57\"/\"kube-root-ca.crt\"" Jul 20 18:29:22.505166 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.505088 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9b1a57\"/\"default-dockercfg-sh88p\"" Jul 20 18:29:22.505166 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.505129 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9b1a57\"/\"test-model-9b1a57-kserve-self-signed-certs\"" Jul 20 18:29:22.511203 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.511181 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5"] Jul 20 18:29:22.600866 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.600836 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-home\") pod \"test-model-9b1a57-kserve-86457857bc-g85p5\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.600866 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.600871 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-tmp-dir\") pod \"test-model-9b1a57-kserve-86457857bc-g85p5\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.601075 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.600905 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-dshm\") pod \"test-model-9b1a57-kserve-86457857bc-g85p5\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.601075 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.600929 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-model-cache\") pod \"test-model-9b1a57-kserve-86457857bc-g85p5\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.601075 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.600951 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0853ebb7-8b20-4f56-8d08-1dbfd557604a-tls-certs\") pod \"test-model-9b1a57-kserve-86457857bc-g85p5\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.601075 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.600975 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57srv\" (UniqueName: \"kubernetes.io/projected/0853ebb7-8b20-4f56-8d08-1dbfd557604a-kube-api-access-57srv\") pod \"test-model-9b1a57-kserve-86457857bc-g85p5\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.701962 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.701929 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-home\") pod \"test-model-9b1a57-kserve-86457857bc-g85p5\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.701962 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.701965 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-tmp-dir\") pod \"test-model-9b1a57-kserve-86457857bc-g85p5\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.702199 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.701986 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-dshm\") pod \"test-model-9b1a57-kserve-86457857bc-g85p5\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.702199 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.702004 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-model-cache\") pod \"test-model-9b1a57-kserve-86457857bc-g85p5\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.702199 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.702025 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0853ebb7-8b20-4f56-8d08-1dbfd557604a-tls-certs\") pod \"test-model-9b1a57-kserve-86457857bc-g85p5\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.702199 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.702058 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57srv\" (UniqueName: \"kubernetes.io/projected/0853ebb7-8b20-4f56-8d08-1dbfd557604a-kube-api-access-57srv\") pod \"test-model-9b1a57-kserve-86457857bc-g85p5\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.702445 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.702376 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-home\") pod \"test-model-9b1a57-kserve-86457857bc-g85p5\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.702445 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.702431 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-tmp-dir\") pod \"test-model-9b1a57-kserve-86457857bc-g85p5\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.702533 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.702491 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-model-cache\") pod \"test-model-9b1a57-kserve-86457857bc-g85p5\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.704242 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.704222 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-dshm\") pod \"test-model-9b1a57-kserve-86457857bc-g85p5\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.704496 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.704479 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0853ebb7-8b20-4f56-8d08-1dbfd557604a-tls-certs\") pod \"test-model-9b1a57-kserve-86457857bc-g85p5\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.710375 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.710352 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57srv\" (UniqueName: \"kubernetes.io/projected/0853ebb7-8b20-4f56-8d08-1dbfd557604a-kube-api-access-57srv\") pod \"test-model-9b1a57-kserve-86457857bc-g85p5\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.810795 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.810677 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:22.955351 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:22.955328 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5"] Jul 20 18:29:22.957059 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:29:22.957027 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0853ebb7_8b20_4f56_8d08_1dbfd557604a.slice/crio-17169df599c9258661c6e4e613003ad9ad5175d5da9c92129b4c870a733a2f0f WatchSource:0}: Error finding container 17169df599c9258661c6e4e613003ad9ad5175d5da9c92129b4c870a733a2f0f: Status 404 returned error can't find the container with id 17169df599c9258661c6e4e613003ad9ad5175d5da9c92129b4c870a733a2f0f Jul 20 18:29:23.583024 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:23.582982 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" event={"ID":"0853ebb7-8b20-4f56-8d08-1dbfd557604a","Type":"ContainerStarted","Data":"8c38811cc84c9a95888a165ebe40c2fd8dcda39c273216c2b7cc9565cbc087f1"} Jul 20 18:29:23.583024 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:23.583032 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" event={"ID":"0853ebb7-8b20-4f56-8d08-1dbfd557604a","Type":"ContainerStarted","Data":"17169df599c9258661c6e4e613003ad9ad5175d5da9c92129b4c870a733a2f0f"} Jul 20 18:29:26.651190 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:26.651135 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" podStartSLOduration=4.651118717 podStartE2EDuration="4.651118717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:29:23.600689948 +0000 UTC m=+2471.972478547" watchObservedRunningTime="2026-07-20 18:29:26.651118717 +0000 UTC m=+2475.022907290" Jul 20 18:29:26.651878 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:26.651860 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:29:32.811697 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:32.811657 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:32.812138 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:32.811720 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:32.824164 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:32.824135 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:33.651669 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:33.651638 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:29:37.047618 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:37.047582 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:29:52.257631 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:29:52.257595 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:30:06.153143 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:06.153061 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:30:29.759089 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:29.759055 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5"] Jul 20 18:30:34.355609 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:34.355568 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:30:44.779104 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:44.779054 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" podUID="0853ebb7-8b20-4f56-8d08-1dbfd557604a" containerName="main" containerID="cri-o://8c38811cc84c9a95888a165ebe40c2fd8dcda39c273216c2b7cc9565cbc087f1" gracePeriod=45 Jul 20 18:30:45.006031 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.006002 2583 generic.go:356] "Generic (PLEG): container finished" podID="0853ebb7-8b20-4f56-8d08-1dbfd557604a" containerID="8c38811cc84c9a95888a165ebe40c2fd8dcda39c273216c2b7cc9565cbc087f1" exitCode=0 Jul 20 18:30:45.006197 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.006076 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" event={"ID":"0853ebb7-8b20-4f56-8d08-1dbfd557604a","Type":"ContainerDied","Data":"8c38811cc84c9a95888a165ebe40c2fd8dcda39c273216c2b7cc9565cbc087f1"} Jul 20 18:30:45.032635 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.032574 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:30:45.099783 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.099718 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-dshm\") pod \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " Jul 20 18:30:45.099952 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.099821 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-tmp-dir\") pod \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " Jul 20 18:30:45.099952 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.099865 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-home\") pod \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " Jul 20 18:30:45.099952 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.099883 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57srv\" (UniqueName: \"kubernetes.io/projected/0853ebb7-8b20-4f56-8d08-1dbfd557604a-kube-api-access-57srv\") pod \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " Jul 20 18:30:45.099952 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.099931 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0853ebb7-8b20-4f56-8d08-1dbfd557604a-tls-certs\") pod \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " Jul 20 18:30:45.100131 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.099965 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-model-cache\") pod \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\" (UID: \"0853ebb7-8b20-4f56-8d08-1dbfd557604a\") " Jul 20 18:30:45.100131 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.100098 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0853ebb7-8b20-4f56-8d08-1dbfd557604a" (UID: "0853ebb7-8b20-4f56-8d08-1dbfd557604a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:30:45.100131 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.100108 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-home" (OuterVolumeSpecName: "home") pod "0853ebb7-8b20-4f56-8d08-1dbfd557604a" (UID: "0853ebb7-8b20-4f56-8d08-1dbfd557604a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:30:45.100292 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.100231 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-tmp-dir\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:30:45.100292 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.100251 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-home\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:30:45.100292 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.100280 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-model-cache" (OuterVolumeSpecName: "model-cache") pod "0853ebb7-8b20-4f56-8d08-1dbfd557604a" (UID: "0853ebb7-8b20-4f56-8d08-1dbfd557604a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:30:45.102012 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.101992 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-dshm" (OuterVolumeSpecName: "dshm") pod "0853ebb7-8b20-4f56-8d08-1dbfd557604a" (UID: "0853ebb7-8b20-4f56-8d08-1dbfd557604a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:30:45.102104 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.102055 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0853ebb7-8b20-4f56-8d08-1dbfd557604a-kube-api-access-57srv" (OuterVolumeSpecName: "kube-api-access-57srv") pod "0853ebb7-8b20-4f56-8d08-1dbfd557604a" (UID: "0853ebb7-8b20-4f56-8d08-1dbfd557604a"). InnerVolumeSpecName "kube-api-access-57srv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:30:45.102104 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.102085 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0853ebb7-8b20-4f56-8d08-1dbfd557604a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0853ebb7-8b20-4f56-8d08-1dbfd557604a" (UID: "0853ebb7-8b20-4f56-8d08-1dbfd557604a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 18:30:45.200994 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.200960 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-dshm\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:30:45.200994 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.200992 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57srv\" (UniqueName: \"kubernetes.io/projected/0853ebb7-8b20-4f56-8d08-1dbfd557604a-kube-api-access-57srv\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:30:45.201200 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.201005 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0853ebb7-8b20-4f56-8d08-1dbfd557604a-tls-certs\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:30:45.201200 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:45.201015 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0853ebb7-8b20-4f56-8d08-1dbfd557604a-model-cache\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:30:46.014075 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:46.014040 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" event={"ID":"0853ebb7-8b20-4f56-8d08-1dbfd557604a","Type":"ContainerDied","Data":"17169df599c9258661c6e4e613003ad9ad5175d5da9c92129b4c870a733a2f0f"} Jul 20 18:30:46.014455 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:46.014093 2583 scope.go:117] "RemoveContainer" containerID="8c38811cc84c9a95888a165ebe40c2fd8dcda39c273216c2b7cc9565cbc087f1" Jul 20 18:30:46.014455 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:46.014089 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5" Jul 20 18:30:46.039114 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:46.039090 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5"] Jul 20 18:30:46.043394 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:46.043371 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9b1a57/test-model-9b1a57-kserve-86457857bc-g85p5"] Jul 20 18:30:46.171897 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:46.171865 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0853ebb7-8b20-4f56-8d08-1dbfd557604a" path="/var/lib/kubelet/pods/0853ebb7-8b20-4f56-8d08-1dbfd557604a/volumes" Jul 20 18:30:46.855241 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:46.855207 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:30:53.150273 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:53.150237 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7"] Jul 20 18:30:53.150698 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:53.150430 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" podUID="7d753058-9ebc-4f57-89a7-5347c2013fb8" containerName="istio-proxy" containerID="cri-o://6ebb459de3cd9eb50e5cb3dc6b7ed69d7638474d4f01fc020196172e7794b43b" gracePeriod=30 Jul 20 18:30:58.398246 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.398218 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:30:58.512895 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.512843 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-envoy\") pod \"7d753058-9ebc-4f57-89a7-5347c2013fb8\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " Jul 20 18:30:58.513083 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.512910 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-workload-socket\") pod \"7d753058-9ebc-4f57-89a7-5347c2013fb8\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " Jul 20 18:30:58.513083 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.512957 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-token\") pod \"7d753058-9ebc-4f57-89a7-5347c2013fb8\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " Jul 20 18:30:58.513083 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.512983 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-workload-certs\") pod \"7d753058-9ebc-4f57-89a7-5347c2013fb8\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " Jul 20 18:30:58.513083 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.513018 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-credential-socket\") pod \"7d753058-9ebc-4f57-89a7-5347c2013fb8\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " Jul 20 18:30:58.513263 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.513174 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d753058-9ebc-4f57-89a7-5347c2013fb8-istiod-ca-cert\") pod \"7d753058-9ebc-4f57-89a7-5347c2013fb8\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " Jul 20 18:30:58.513322 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.513183 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7d753058-9ebc-4f57-89a7-5347c2013fb8" (UID: "7d753058-9ebc-4f57-89a7-5347c2013fb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:30:58.513322 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.513262 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pq6v\" (UniqueName: \"kubernetes.io/projected/7d753058-9ebc-4f57-89a7-5347c2013fb8-kube-api-access-5pq6v\") pod \"7d753058-9ebc-4f57-89a7-5347c2013fb8\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " Jul 20 18:30:58.513432 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.513309 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7d753058-9ebc-4f57-89a7-5347c2013fb8" (UID: "7d753058-9ebc-4f57-89a7-5347c2013fb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:30:58.513432 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.513354 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-podinfo\") pod \"7d753058-9ebc-4f57-89a7-5347c2013fb8\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " Jul 20 18:30:58.513432 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.513352 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7d753058-9ebc-4f57-89a7-5347c2013fb8" (UID: "7d753058-9ebc-4f57-89a7-5347c2013fb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:30:58.513432 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.513397 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-data\") pod \"7d753058-9ebc-4f57-89a7-5347c2013fb8\" (UID: \"7d753058-9ebc-4f57-89a7-5347c2013fb8\") " Jul 20 18:30:58.513619 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.513454 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d753058-9ebc-4f57-89a7-5347c2013fb8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7d753058-9ebc-4f57-89a7-5347c2013fb8" (UID: "7d753058-9ebc-4f57-89a7-5347c2013fb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 18:30:58.513682 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.513638 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-workload-socket\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:30:58.513682 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.513658 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-workload-certs\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:30:58.513682 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.513671 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-credential-socket\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:30:58.513845 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.513686 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d753058-9ebc-4f57-89a7-5347c2013fb8-istiod-ca-cert\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:30:58.514573 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.514548 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-data" (OuterVolumeSpecName: "istio-data") pod "7d753058-9ebc-4f57-89a7-5347c2013fb8" (UID: "7d753058-9ebc-4f57-89a7-5347c2013fb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:30:58.515102 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.515083 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7d753058-9ebc-4f57-89a7-5347c2013fb8" (UID: "7d753058-9ebc-4f57-89a7-5347c2013fb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:30:58.515518 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.515503 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7d753058-9ebc-4f57-89a7-5347c2013fb8" (UID: "7d753058-9ebc-4f57-89a7-5347c2013fb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 18:30:58.515645 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.515624 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-token" (OuterVolumeSpecName: "istio-token") pod "7d753058-9ebc-4f57-89a7-5347c2013fb8" (UID: "7d753058-9ebc-4f57-89a7-5347c2013fb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:30:58.515702 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.515639 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d753058-9ebc-4f57-89a7-5347c2013fb8-kube-api-access-5pq6v" (OuterVolumeSpecName: "kube-api-access-5pq6v") pod "7d753058-9ebc-4f57-89a7-5347c2013fb8" (UID: "7d753058-9ebc-4f57-89a7-5347c2013fb8"). InnerVolumeSpecName "kube-api-access-5pq6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:30:58.614911 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.614861 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-token\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:30:58.614911 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.614904 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pq6v\" (UniqueName: \"kubernetes.io/projected/7d753058-9ebc-4f57-89a7-5347c2013fb8-kube-api-access-5pq6v\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:30:58.614911 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.614914 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-podinfo\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:30:58.614911 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.614925 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-data\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:30:58.614911 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:58.614936 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d753058-9ebc-4f57-89a7-5347c2013fb8-istio-envoy\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:30:59.082240 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:59.082203 2583 generic.go:356] "Generic (PLEG): container finished" podID="7d753058-9ebc-4f57-89a7-5347c2013fb8" containerID="6ebb459de3cd9eb50e5cb3dc6b7ed69d7638474d4f01fc020196172e7794b43b" exitCode=0 Jul 20 18:30:59.082449 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:59.082253 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" event={"ID":"7d753058-9ebc-4f57-89a7-5347c2013fb8","Type":"ContainerDied","Data":"6ebb459de3cd9eb50e5cb3dc6b7ed69d7638474d4f01fc020196172e7794b43b"} Jul 20 18:30:59.082449 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:59.082303 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" event={"ID":"7d753058-9ebc-4f57-89a7-5347c2013fb8","Type":"ContainerDied","Data":"6fc6dfc5c6ba253234f7c007cd83dda5d830e95baff2768e128183a39bdd5188"} Jul 20 18:30:59.082449 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:59.082325 2583 scope.go:117] "RemoveContainer" containerID="6ebb459de3cd9eb50e5cb3dc6b7ed69d7638474d4f01fc020196172e7794b43b" Jul 20 18:30:59.082449 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:59.082263 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7" Jul 20 18:30:59.091805 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:59.091788 2583 scope.go:117] "RemoveContainer" containerID="699e55ed11fa5bd04b40dc257aaefd48a5daa81557bfc2fcf147ed374ce5d758" Jul 20 18:30:59.099252 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:59.099236 2583 scope.go:117] "RemoveContainer" containerID="6ebb459de3cd9eb50e5cb3dc6b7ed69d7638474d4f01fc020196172e7794b43b" Jul 20 18:30:59.099507 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:30:59.099489 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ebb459de3cd9eb50e5cb3dc6b7ed69d7638474d4f01fc020196172e7794b43b\": container with ID starting with 6ebb459de3cd9eb50e5cb3dc6b7ed69d7638474d4f01fc020196172e7794b43b not found: ID does not exist" containerID="6ebb459de3cd9eb50e5cb3dc6b7ed69d7638474d4f01fc020196172e7794b43b" Jul 20 18:30:59.099555 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:59.099517 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ebb459de3cd9eb50e5cb3dc6b7ed69d7638474d4f01fc020196172e7794b43b"} err="failed to get container status \"6ebb459de3cd9eb50e5cb3dc6b7ed69d7638474d4f01fc020196172e7794b43b\": rpc error: code = NotFound desc = could not find container \"6ebb459de3cd9eb50e5cb3dc6b7ed69d7638474d4f01fc020196172e7794b43b\": container with ID starting with 6ebb459de3cd9eb50e5cb3dc6b7ed69d7638474d4f01fc020196172e7794b43b not found: ID does not exist" Jul 20 18:30:59.099555 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:59.099534 2583 scope.go:117] "RemoveContainer" containerID="699e55ed11fa5bd04b40dc257aaefd48a5daa81557bfc2fcf147ed374ce5d758" Jul 20 18:30:59.099789 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:30:59.099769 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"699e55ed11fa5bd04b40dc257aaefd48a5daa81557bfc2fcf147ed374ce5d758\": container with ID starting with 699e55ed11fa5bd04b40dc257aaefd48a5daa81557bfc2fcf147ed374ce5d758 not found: ID does not exist" containerID="699e55ed11fa5bd04b40dc257aaefd48a5daa81557bfc2fcf147ed374ce5d758" Jul 20 18:30:59.099837 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:59.099799 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"699e55ed11fa5bd04b40dc257aaefd48a5daa81557bfc2fcf147ed374ce5d758"} err="failed to get container status \"699e55ed11fa5bd04b40dc257aaefd48a5daa81557bfc2fcf147ed374ce5d758\": rpc error: code = NotFound desc = could not find container \"699e55ed11fa5bd04b40dc257aaefd48a5daa81557bfc2fcf147ed374ce5d758\": container with ID starting with 699e55ed11fa5bd04b40dc257aaefd48a5daa81557bfc2fcf147ed374ce5d758 not found: ID does not exist" Jul 20 18:30:59.106086 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:59.106067 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7"] Jul 20 18:30:59.111175 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:30:59.111155 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9b1a57-openshift-default-759cc468d7-xpmc7"] Jul 20 18:31:00.170272 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:00.170232 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d753058-9ebc-4f57-89a7-5347c2013fb8" path="/var/lib/kubelet/pods/7d753058-9ebc-4f57-89a7-5347c2013fb8/volumes" Jul 20 18:31:07.354404 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:07.354370 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:31:16.228997 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.228943 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5"] Jul 20 18:31:16.229457 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.229309 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d753058-9ebc-4f57-89a7-5347c2013fb8" containerName="istio-proxy" Jul 20 18:31:16.229457 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.229321 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d753058-9ebc-4f57-89a7-5347c2013fb8" containerName="istio-proxy" Jul 20 18:31:16.229457 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.229336 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d753058-9ebc-4f57-89a7-5347c2013fb8" containerName="istio-proxy" Jul 20 18:31:16.229457 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.229362 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d753058-9ebc-4f57-89a7-5347c2013fb8" containerName="istio-proxy" Jul 20 18:31:16.229457 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.229372 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0853ebb7-8b20-4f56-8d08-1dbfd557604a" containerName="main" Jul 20 18:31:16.229457 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.229377 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0853ebb7-8b20-4f56-8d08-1dbfd557604a" containerName="main" Jul 20 18:31:16.229457 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.229437 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d753058-9ebc-4f57-89a7-5347c2013fb8" containerName="istio-proxy" Jul 20 18:31:16.229457 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.229446 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d753058-9ebc-4f57-89a7-5347c2013fb8" containerName="istio-proxy" Jul 20 18:31:16.229457 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.229458 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0853ebb7-8b20-4f56-8d08-1dbfd557604a" containerName="main" Jul 20 18:31:16.234200 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.234180 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.236700 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.236672 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f9ae33-openshift-default-dockercfg-ctdgc\"" Jul 20 18:31:16.246511 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.246491 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5"] Jul 20 18:31:16.267643 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.267601 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-workload-certs\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.267860 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.267650 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-credential-socket\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.267860 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.267767 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-envoy\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.267860 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.267795 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-data\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.268060 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.267875 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-workload-socket\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.268060 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.267935 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-podinfo\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.268060 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.267978 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-token\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.268060 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.268008 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5103e72-0afe-4f54-9280-f0d1adf301a2-istiod-ca-cert\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.268060 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.268035 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh8xb\" (UniqueName: \"kubernetes.io/projected/c5103e72-0afe-4f54-9280-f0d1adf301a2-kube-api-access-zh8xb\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.368453 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.368412 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-workload-socket\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.368453 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.368455 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-podinfo\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.368679 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.368474 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-token\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.368679 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.368590 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5103e72-0afe-4f54-9280-f0d1adf301a2-istiod-ca-cert\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.368679 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.368629 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8xb\" (UniqueName: \"kubernetes.io/projected/c5103e72-0afe-4f54-9280-f0d1adf301a2-kube-api-access-zh8xb\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.368888 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.368678 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-workload-certs\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.368888 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.368753 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-credential-socket\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.368888 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.368818 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-envoy\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.368888 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.368847 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-data\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.368888 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.368857 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-workload-socket\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.369179 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.369116 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-workload-certs\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.369245 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.369206 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-credential-socket\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.369245 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.369216 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-data\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.369379 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.369360 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5103e72-0afe-4f54-9280-f0d1adf301a2-istiod-ca-cert\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.370967 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.370948 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-envoy\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.371081 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.371063 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-podinfo\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.379512 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.379488 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh8xb\" (UniqueName: \"kubernetes.io/projected/c5103e72-0afe-4f54-9280-f0d1adf301a2-kube-api-access-zh8xb\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.379512 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.379510 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-token\") pod \"e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.546049 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.545954 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:16.687846 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.687821 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5"] Jul 20 18:31:16.689209 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:31:16.689184 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5103e72_0afe_4f54_9280_f0d1adf301a2.slice/crio-f6a6e5e27ce33ec6514eda65950d0d0cc58695b5caa37660aec2ea6b7255cc9c WatchSource:0}: Error finding container f6a6e5e27ce33ec6514eda65950d0d0cc58695b5caa37660aec2ea6b7255cc9c: Status 404 returned error can't find the container with id f6a6e5e27ce33ec6514eda65950d0d0cc58695b5caa37660aec2ea6b7255cc9c Jul 20 18:31:16.691223 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.691194 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:31:16.691294 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.691258 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:31:16.691294 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:16.691286 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:31:17.175701 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:17.175661 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" event={"ID":"c5103e72-0afe-4f54-9280-f0d1adf301a2","Type":"ContainerStarted","Data":"77ddb75b4173d0d8ba49d7f99c490ed8648bf1449e8de086b58629461b650b24"} Jul 20 18:31:17.175909 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:17.175709 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" event={"ID":"c5103e72-0afe-4f54-9280-f0d1adf301a2","Type":"ContainerStarted","Data":"f6a6e5e27ce33ec6514eda65950d0d0cc58695b5caa37660aec2ea6b7255cc9c"} Jul 20 18:31:17.546508 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:17.546469 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:17.551329 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:17.551304 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:17.575431 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:17.575353 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" podStartSLOduration=1.575336933 podStartE2EDuration="1.575336933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:31:17.220434009 +0000 UTC m=+2585.592222585" watchObservedRunningTime="2026-07-20 18:31:17.575336933 +0000 UTC m=+2585.947125486" Jul 20 18:31:18.180667 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:18.180636 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:18.181783 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:18.181763 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:22.255833 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:22.255794 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:31:24.696338 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.696307 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg"] Jul 20 18:31:24.699701 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.699683 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.704999 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.704980 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9c6cfa-openshift-default-dockercfg-58t6q\"" Jul 20 18:31:24.721712 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.721676 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg"] Jul 20 18:31:24.741336 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.741291 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-token\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.741672 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.741630 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-workload-socket\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.741870 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.741853 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-envoy\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.742044 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.742027 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-data\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.742197 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.742182 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-podinfo\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.742344 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.742329 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istiod-ca-cert\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.742500 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.742483 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7cvr\" (UniqueName: \"kubernetes.io/projected/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-kube-api-access-f7cvr\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.742673 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.742654 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-credential-socket\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.742851 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.742834 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-workload-certs\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.844430 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.844392 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-credential-socket\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.844430 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.844434 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-workload-certs\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.844674 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.844465 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-token\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.844674 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.844601 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-workload-socket\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.844674 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.844648 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-envoy\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.844867 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.844704 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-data\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.844867 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.844763 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-podinfo\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.844867 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.844789 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istiod-ca-cert\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.844867 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.844826 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7cvr\" (UniqueName: \"kubernetes.io/projected/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-kube-api-access-f7cvr\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.844867 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.844827 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-workload-certs\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.845131 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.844949 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-workload-socket\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.845131 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.845082 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-data\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.845244 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.845169 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-credential-socket\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.845466 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.845447 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istiod-ca-cert\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.846952 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.846930 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-envoy\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.847334 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.847312 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-podinfo\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.855940 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.855919 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-token\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:24.856282 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:24.856262 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7cvr\" (UniqueName: \"kubernetes.io/projected/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-kube-api-access-f7cvr\") pod \"e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:25.010945 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:25.010858 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:25.193240 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:25.193215 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg"] Jul 20 18:31:25.194548 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:31:25.194522 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dccab46_f31b_4a1b_ba97_d6fed4f58f16.slice/crio-6031cd860401bedbee7506e50accb1afc1777092f19f134f46b831ec015f2cac WatchSource:0}: Error finding container 6031cd860401bedbee7506e50accb1afc1777092f19f134f46b831ec015f2cac: Status 404 returned error can't find the container with id 6031cd860401bedbee7506e50accb1afc1777092f19f134f46b831ec015f2cac Jul 20 18:31:25.196975 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:25.196940 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:31:25.197063 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:25.197001 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:31:25.197063 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:25.197030 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:31:25.245406 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:25.245363 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" event={"ID":"9dccab46-f31b-4a1b-ba97-d6fed4f58f16","Type":"ContainerStarted","Data":"6031cd860401bedbee7506e50accb1afc1777092f19f134f46b831ec015f2cac"} Jul 20 18:31:26.252983 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:26.252939 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" event={"ID":"9dccab46-f31b-4a1b-ba97-d6fed4f58f16","Type":"ContainerStarted","Data":"e502f03c9af293f1637bbc614dfc6d66f56b7722a30941cb2735e0750aceac38"} Jul 20 18:31:26.276604 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:26.276527 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" podStartSLOduration=2.276504598 podStartE2EDuration="2.276504598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:31:26.274232187 +0000 UTC m=+2594.646020761" watchObservedRunningTime="2026-07-20 18:31:26.276504598 +0000 UTC m=+2594.648293172" Jul 20 18:31:27.011564 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:27.011527 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:27.023169 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:27.023132 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:27.257749 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:27.257699 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:27.258841 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:27.258822 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:31.281011 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:31.280980 2583 generic.go:356] "Generic (PLEG): container finished" podID="c5103e72-0afe-4f54-9280-f0d1adf301a2" containerID="77ddb75b4173d0d8ba49d7f99c490ed8648bf1449e8de086b58629461b650b24" exitCode=0 Jul 20 18:31:31.281429 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:31.281056 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" event={"ID":"c5103e72-0afe-4f54-9280-f0d1adf301a2","Type":"ContainerDied","Data":"77ddb75b4173d0d8ba49d7f99c490ed8648bf1449e8de086b58629461b650b24"} Jul 20 18:31:31.281472 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:31.281458 2583 scope.go:117] "RemoveContainer" containerID="77ddb75b4173d0d8ba49d7f99c490ed8648bf1449e8de086b58629461b650b24" Jul 20 18:31:31.282482 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:31.282418 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:31:31.282534 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:31.282521 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:31:31.282574 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:31.282556 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:31:32.288234 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:32.288184 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" event={"ID":"c5103e72-0afe-4f54-9280-f0d1adf301a2","Type":"ContainerStarted","Data":"f1cfd578b39478efb7545d742eb700918072fe5a713c004f27fc873b77b057c6"} Jul 20 18:31:32.546346 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:32.546227 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:32.547444 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:32.547421 2583 patch_prober.go:28] interesting pod/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 20 18:31:32.547550 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:32.547513 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" podUID="c5103e72-0afe-4f54-9280-f0d1adf301a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 20 18:31:33.181555 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:33.181522 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:33.547300 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:33.547196 2583 patch_prober.go:28] interesting pod/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 20 18:31:33.547300 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:33.547260 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" podUID="c5103e72-0afe-4f54-9280-f0d1adf301a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 20 18:31:34.546623 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:34.546582 2583 patch_prober.go:28] interesting pod/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 20 18:31:34.546849 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:34.546655 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" podUID="c5103e72-0afe-4f54-9280-f0d1adf301a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 20 18:31:35.546860 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:35.546824 2583 patch_prober.go:28] interesting pod/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 20 18:31:35.547268 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:35.546893 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" podUID="c5103e72-0afe-4f54-9280-f0d1adf301a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 20 18:31:36.551299 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:36.551267 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:36.552209 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:36.552190 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:31:37.158948 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:37.158916 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:31:45.364366 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:45.364266 2583 generic.go:356] "Generic (PLEG): container finished" podID="9dccab46-f31b-4a1b-ba97-d6fed4f58f16" containerID="e502f03c9af293f1637bbc614dfc6d66f56b7722a30941cb2735e0750aceac38" exitCode=0 Jul 20 18:31:45.364366 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:45.364337 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" event={"ID":"9dccab46-f31b-4a1b-ba97-d6fed4f58f16","Type":"ContainerDied","Data":"e502f03c9af293f1637bbc614dfc6d66f56b7722a30941cb2735e0750aceac38"} Jul 20 18:31:45.364860 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:45.364755 2583 scope.go:117] "RemoveContainer" containerID="e502f03c9af293f1637bbc614dfc6d66f56b7722a30941cb2735e0750aceac38" Jul 20 18:31:45.365800 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:45.365771 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:31:45.365914 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:45.365852 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:31:45.365914 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:45.365903 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:31:46.371528 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:46.371495 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" event={"ID":"9dccab46-f31b-4a1b-ba97-d6fed4f58f16","Type":"ContainerStarted","Data":"14bb767d8917dd7a868e6d65403f174b05b78f984d6ad3e7b2e502c93c6ce0d9"} Jul 20 18:31:47.011963 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:47.011923 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:47.012173 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:47.012119 2583 patch_prober.go:28] interesting pod/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 20 18:31:47.012173 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:47.012161 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" podUID="9dccab46-f31b-4a1b-ba97-d6fed4f58f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 20 18:31:48.011742 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:48.011699 2583 patch_prober.go:28] interesting pod/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 20 18:31:48.012230 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:48.011789 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" podUID="9dccab46-f31b-4a1b-ba97-d6fed4f58f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 20 18:31:49.011687 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:49.011647 2583 patch_prober.go:28] interesting pod/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 20 18:31:49.011950 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:49.011708 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" podUID="9dccab46-f31b-4a1b-ba97-d6fed4f58f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 20 18:31:50.011450 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:50.011410 2583 patch_prober.go:28] interesting pod/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 20 18:31:50.011642 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:50.011467 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" podUID="9dccab46-f31b-4a1b-ba97-d6fed4f58f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 20 18:31:51.015382 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:51.015342 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:51.015830 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:51.015633 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:31:51.016184 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:31:51.016164 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:32:02.754101 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:02.754050 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:32:07.581018 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:07.580982 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5"] Jul 20 18:32:07.581445 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:07.581235 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" podUID="c5103e72-0afe-4f54-9280-f0d1adf301a2" containerName="istio-proxy" containerID="cri-o://f1cfd578b39478efb7545d742eb700918072fe5a713c004f27fc873b77b057c6" gracePeriod=30 Jul 20 18:32:12.828065 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.828041 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:32:12.878160 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.878075 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-envoy\") pod \"c5103e72-0afe-4f54-9280-f0d1adf301a2\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " Jul 20 18:32:12.878160 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.878111 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5103e72-0afe-4f54-9280-f0d1adf301a2-istiod-ca-cert\") pod \"c5103e72-0afe-4f54-9280-f0d1adf301a2\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " Jul 20 18:32:12.878160 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.878143 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh8xb\" (UniqueName: \"kubernetes.io/projected/c5103e72-0afe-4f54-9280-f0d1adf301a2-kube-api-access-zh8xb\") pod \"c5103e72-0afe-4f54-9280-f0d1adf301a2\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " Jul 20 18:32:12.878448 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.878170 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-token\") pod \"c5103e72-0afe-4f54-9280-f0d1adf301a2\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " Jul 20 18:32:12.878448 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.878208 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-credential-socket\") pod \"c5103e72-0afe-4f54-9280-f0d1adf301a2\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " Jul 20 18:32:12.878448 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.878275 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-workload-certs\") pod \"c5103e72-0afe-4f54-9280-f0d1adf301a2\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " Jul 20 18:32:12.878448 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.878299 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-podinfo\") pod \"c5103e72-0afe-4f54-9280-f0d1adf301a2\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " Jul 20 18:32:12.878448 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.878327 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-data\") pod \"c5103e72-0afe-4f54-9280-f0d1adf301a2\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " Jul 20 18:32:12.878448 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.878352 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-workload-socket\") pod \"c5103e72-0afe-4f54-9280-f0d1adf301a2\" (UID: \"c5103e72-0afe-4f54-9280-f0d1adf301a2\") " Jul 20 18:32:12.878814 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.878548 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c5103e72-0afe-4f54-9280-f0d1adf301a2" (UID: "c5103e72-0afe-4f54-9280-f0d1adf301a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:32:12.878814 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.878578 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5103e72-0afe-4f54-9280-f0d1adf301a2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c5103e72-0afe-4f54-9280-f0d1adf301a2" (UID: "c5103e72-0afe-4f54-9280-f0d1adf301a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 18:32:12.878814 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.878668 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-workload-certs\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:32:12.878814 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.878787 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c5103e72-0afe-4f54-9280-f0d1adf301a2" (UID: "c5103e72-0afe-4f54-9280-f0d1adf301a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:32:12.879032 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.878818 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c5103e72-0afe-4f54-9280-f0d1adf301a2" (UID: "c5103e72-0afe-4f54-9280-f0d1adf301a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:32:12.879480 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.879451 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-data" (OuterVolumeSpecName: "istio-data") pod "c5103e72-0afe-4f54-9280-f0d1adf301a2" (UID: "c5103e72-0afe-4f54-9280-f0d1adf301a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:32:12.880430 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.880407 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c5103e72-0afe-4f54-9280-f0d1adf301a2" (UID: "c5103e72-0afe-4f54-9280-f0d1adf301a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:32:12.880514 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.880412 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5103e72-0afe-4f54-9280-f0d1adf301a2-kube-api-access-zh8xb" (OuterVolumeSpecName: "kube-api-access-zh8xb") pod "c5103e72-0afe-4f54-9280-f0d1adf301a2" (UID: "c5103e72-0afe-4f54-9280-f0d1adf301a2"). InnerVolumeSpecName "kube-api-access-zh8xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:32:12.880514 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.880433 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-token" (OuterVolumeSpecName: "istio-token") pod "c5103e72-0afe-4f54-9280-f0d1adf301a2" (UID: "c5103e72-0afe-4f54-9280-f0d1adf301a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:32:12.881053 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.881028 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c5103e72-0afe-4f54-9280-f0d1adf301a2" (UID: "c5103e72-0afe-4f54-9280-f0d1adf301a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 18:32:12.979236 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.979190 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-credential-socket\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:32:12.979236 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.979236 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-podinfo\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:32:12.979236 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.979250 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-data\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:32:12.979504 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.979263 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-workload-socket\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:32:12.979504 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.979274 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-envoy\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:32:12.979504 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.979287 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5103e72-0afe-4f54-9280-f0d1adf301a2-istiod-ca-cert\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:32:12.979504 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.979300 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh8xb\" (UniqueName: \"kubernetes.io/projected/c5103e72-0afe-4f54-9280-f0d1adf301a2-kube-api-access-zh8xb\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:32:12.979504 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:12.979311 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5103e72-0afe-4f54-9280-f0d1adf301a2-istio-token\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:32:13.509218 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:13.509183 2583 generic.go:356] "Generic (PLEG): container finished" podID="c5103e72-0afe-4f54-9280-f0d1adf301a2" containerID="f1cfd578b39478efb7545d742eb700918072fe5a713c004f27fc873b77b057c6" exitCode=0 Jul 20 18:32:13.509401 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:13.509234 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" Jul 20 18:32:13.509401 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:13.509251 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" event={"ID":"c5103e72-0afe-4f54-9280-f0d1adf301a2","Type":"ContainerDied","Data":"f1cfd578b39478efb7545d742eb700918072fe5a713c004f27fc873b77b057c6"} Jul 20 18:32:13.509401 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:13.509299 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5" event={"ID":"c5103e72-0afe-4f54-9280-f0d1adf301a2","Type":"ContainerDied","Data":"f6a6e5e27ce33ec6514eda65950d0d0cc58695b5caa37660aec2ea6b7255cc9c"} Jul 20 18:32:13.509401 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:13.509321 2583 scope.go:117] "RemoveContainer" containerID="f1cfd578b39478efb7545d742eb700918072fe5a713c004f27fc873b77b057c6" Jul 20 18:32:13.519337 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:13.519315 2583 scope.go:117] "RemoveContainer" containerID="77ddb75b4173d0d8ba49d7f99c490ed8648bf1449e8de086b58629461b650b24" Jul 20 18:32:13.529940 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:13.528439 2583 scope.go:117] "RemoveContainer" containerID="f1cfd578b39478efb7545d742eb700918072fe5a713c004f27fc873b77b057c6" Jul 20 18:32:13.529940 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:32:13.528825 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1cfd578b39478efb7545d742eb700918072fe5a713c004f27fc873b77b057c6\": container with ID starting with f1cfd578b39478efb7545d742eb700918072fe5a713c004f27fc873b77b057c6 not found: ID does not exist" containerID="f1cfd578b39478efb7545d742eb700918072fe5a713c004f27fc873b77b057c6" Jul 20 18:32:13.529940 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:13.528857 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1cfd578b39478efb7545d742eb700918072fe5a713c004f27fc873b77b057c6"} err="failed to get container status \"f1cfd578b39478efb7545d742eb700918072fe5a713c004f27fc873b77b057c6\": rpc error: code = NotFound desc = could not find container \"f1cfd578b39478efb7545d742eb700918072fe5a713c004f27fc873b77b057c6\": container with ID starting with f1cfd578b39478efb7545d742eb700918072fe5a713c004f27fc873b77b057c6 not found: ID does not exist" Jul 20 18:32:13.529940 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:13.528881 2583 scope.go:117] "RemoveContainer" containerID="77ddb75b4173d0d8ba49d7f99c490ed8648bf1449e8de086b58629461b650b24" Jul 20 18:32:13.529940 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:32:13.529236 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ddb75b4173d0d8ba49d7f99c490ed8648bf1449e8de086b58629461b650b24\": container with ID starting with 77ddb75b4173d0d8ba49d7f99c490ed8648bf1449e8de086b58629461b650b24 not found: ID does not exist" containerID="77ddb75b4173d0d8ba49d7f99c490ed8648bf1449e8de086b58629461b650b24" Jul 20 18:32:13.529940 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:13.529258 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ddb75b4173d0d8ba49d7f99c490ed8648bf1449e8de086b58629461b650b24"} err="failed to get container status \"77ddb75b4173d0d8ba49d7f99c490ed8648bf1449e8de086b58629461b650b24\": rpc error: code = NotFound desc = could not find container \"77ddb75b4173d0d8ba49d7f99c490ed8648bf1449e8de086b58629461b650b24\": container with ID starting with 77ddb75b4173d0d8ba49d7f99c490ed8648bf1449e8de086b58629461b650b24 not found: ID does not exist" Jul 20 18:32:13.534214 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:13.534192 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5"] Jul 20 18:32:13.538889 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:13.538867 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f9ae33-openshift-default-797d496fc-rrjj5"] Jul 20 18:32:14.169852 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:14.169820 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5103e72-0afe-4f54-9280-f0d1adf301a2" path="/var/lib/kubelet/pods/c5103e72-0afe-4f54-9280-f0d1adf301a2/volumes" Jul 20 18:32:28.647066 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:28.647028 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:32:35.040330 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:35.040297 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg"] Jul 20 18:32:35.040869 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:35.040533 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" podUID="9dccab46-f31b-4a1b-ba97-d6fed4f58f16" containerName="istio-proxy" containerID="cri-o://14bb767d8917dd7a868e6d65403f174b05b78f984d6ad3e7b2e502c93c6ce0d9" gracePeriod=30 Jul 20 18:32:36.018437 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:36.018396 2583 patch_prober.go:28] interesting pod/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 20 18:32:36.018609 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:36.018476 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" podUID="9dccab46-f31b-4a1b-ba97-d6fed4f58f16" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 18:32:40.290909 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.290880 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:32:40.421252 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.421149 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istiod-ca-cert\") pod \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " Jul 20 18:32:40.421252 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.421195 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-credential-socket\") pod \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " Jul 20 18:32:40.421252 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.421216 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-token\") pod \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " Jul 20 18:32:40.421252 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.421233 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-podinfo\") pod \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " Jul 20 18:32:40.421252 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.421254 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-workload-certs\") pod \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " Jul 20 18:32:40.421651 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.421286 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-data\") pod \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " Jul 20 18:32:40.421651 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.421350 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-envoy\") pod \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " Jul 20 18:32:40.421651 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.421388 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-workload-socket\") pod \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " Jul 20 18:32:40.421651 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.421412 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7cvr\" (UniqueName: \"kubernetes.io/projected/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-kube-api-access-f7cvr\") pod \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\" (UID: \"9dccab46-f31b-4a1b-ba97-d6fed4f58f16\") " Jul 20 18:32:40.421651 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.421514 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9dccab46-f31b-4a1b-ba97-d6fed4f58f16" (UID: "9dccab46-f31b-4a1b-ba97-d6fed4f58f16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:32:40.421969 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.421684 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-credential-socket\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:32:40.421969 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.421690 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9dccab46-f31b-4a1b-ba97-d6fed4f58f16" (UID: "9dccab46-f31b-4a1b-ba97-d6fed4f58f16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:32:40.421969 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.421615 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9dccab46-f31b-4a1b-ba97-d6fed4f58f16" (UID: "9dccab46-f31b-4a1b-ba97-d6fed4f58f16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 18:32:40.421969 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.421714 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9dccab46-f31b-4a1b-ba97-d6fed4f58f16" (UID: "9dccab46-f31b-4a1b-ba97-d6fed4f58f16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:32:40.422548 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.422522 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-data" (OuterVolumeSpecName: "istio-data") pod "9dccab46-f31b-4a1b-ba97-d6fed4f58f16" (UID: "9dccab46-f31b-4a1b-ba97-d6fed4f58f16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:32:40.423575 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.423554 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9dccab46-f31b-4a1b-ba97-d6fed4f58f16" (UID: "9dccab46-f31b-4a1b-ba97-d6fed4f58f16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 18:32:40.423658 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.423608 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-kube-api-access-f7cvr" (OuterVolumeSpecName: "kube-api-access-f7cvr") pod "9dccab46-f31b-4a1b-ba97-d6fed4f58f16" (UID: "9dccab46-f31b-4a1b-ba97-d6fed4f58f16"). InnerVolumeSpecName "kube-api-access-f7cvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:32:40.423897 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.423879 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-token" (OuterVolumeSpecName: "istio-token") pod "9dccab46-f31b-4a1b-ba97-d6fed4f58f16" (UID: "9dccab46-f31b-4a1b-ba97-d6fed4f58f16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:32:40.423990 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.423969 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9dccab46-f31b-4a1b-ba97-d6fed4f58f16" (UID: "9dccab46-f31b-4a1b-ba97-d6fed4f58f16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:32:40.522396 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.522357 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-envoy\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:32:40.522396 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.522390 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-workload-socket\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:32:40.522396 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.522403 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7cvr\" (UniqueName: \"kubernetes.io/projected/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-kube-api-access-f7cvr\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:32:40.522648 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.522416 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istiod-ca-cert\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:32:40.522648 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.522428 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-token\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:32:40.522648 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.522439 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-podinfo\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:32:40.522648 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.522449 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-workload-certs\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:32:40.522648 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.522460 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dccab46-f31b-4a1b-ba97-d6fed4f58f16-istio-data\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:32:40.647272 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.647238 2583 generic.go:356] "Generic (PLEG): container finished" podID="9dccab46-f31b-4a1b-ba97-d6fed4f58f16" containerID="14bb767d8917dd7a868e6d65403f174b05b78f984d6ad3e7b2e502c93c6ce0d9" exitCode=0 Jul 20 18:32:40.647471 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.647283 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" Jul 20 18:32:40.647471 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.647327 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" event={"ID":"9dccab46-f31b-4a1b-ba97-d6fed4f58f16","Type":"ContainerDied","Data":"14bb767d8917dd7a868e6d65403f174b05b78f984d6ad3e7b2e502c93c6ce0d9"} Jul 20 18:32:40.647471 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.647384 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg" event={"ID":"9dccab46-f31b-4a1b-ba97-d6fed4f58f16","Type":"ContainerDied","Data":"6031cd860401bedbee7506e50accb1afc1777092f19f134f46b831ec015f2cac"} Jul 20 18:32:40.647471 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.647408 2583 scope.go:117] "RemoveContainer" containerID="14bb767d8917dd7a868e6d65403f174b05b78f984d6ad3e7b2e502c93c6ce0d9" Jul 20 18:32:40.657153 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.657137 2583 scope.go:117] "RemoveContainer" containerID="e502f03c9af293f1637bbc614dfc6d66f56b7722a30941cb2735e0750aceac38" Jul 20 18:32:40.665375 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.665354 2583 scope.go:117] "RemoveContainer" containerID="14bb767d8917dd7a868e6d65403f174b05b78f984d6ad3e7b2e502c93c6ce0d9" Jul 20 18:32:40.665669 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:32:40.665650 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14bb767d8917dd7a868e6d65403f174b05b78f984d6ad3e7b2e502c93c6ce0d9\": container with ID starting with 14bb767d8917dd7a868e6d65403f174b05b78f984d6ad3e7b2e502c93c6ce0d9 not found: ID does not exist" containerID="14bb767d8917dd7a868e6d65403f174b05b78f984d6ad3e7b2e502c93c6ce0d9" Jul 20 18:32:40.665747 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.665677 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14bb767d8917dd7a868e6d65403f174b05b78f984d6ad3e7b2e502c93c6ce0d9"} err="failed to get container status \"14bb767d8917dd7a868e6d65403f174b05b78f984d6ad3e7b2e502c93c6ce0d9\": rpc error: code = NotFound desc = could not find container \"14bb767d8917dd7a868e6d65403f174b05b78f984d6ad3e7b2e502c93c6ce0d9\": container with ID starting with 14bb767d8917dd7a868e6d65403f174b05b78f984d6ad3e7b2e502c93c6ce0d9 not found: ID does not exist" Jul 20 18:32:40.665747 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.665696 2583 scope.go:117] "RemoveContainer" containerID="e502f03c9af293f1637bbc614dfc6d66f56b7722a30941cb2735e0750aceac38" Jul 20 18:32:40.665938 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:32:40.665923 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e502f03c9af293f1637bbc614dfc6d66f56b7722a30941cb2735e0750aceac38\": container with ID starting with e502f03c9af293f1637bbc614dfc6d66f56b7722a30941cb2735e0750aceac38 not found: ID does not exist" containerID="e502f03c9af293f1637bbc614dfc6d66f56b7722a30941cb2735e0750aceac38" Jul 20 18:32:40.665984 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.665941 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e502f03c9af293f1637bbc614dfc6d66f56b7722a30941cb2735e0750aceac38"} err="failed to get container status \"e502f03c9af293f1637bbc614dfc6d66f56b7722a30941cb2735e0750aceac38\": rpc error: code = NotFound desc = could not find container \"e502f03c9af293f1637bbc614dfc6d66f56b7722a30941cb2735e0750aceac38\": container with ID starting with e502f03c9af293f1637bbc614dfc6d66f56b7722a30941cb2735e0750aceac38 not found: ID does not exist" Jul 20 18:32:40.672486 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.672426 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg"] Jul 20 18:32:40.677411 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:40.677388 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9c6cfa-openshift-default-658cd9d8b5-hfwwg"] Jul 20 18:32:42.170463 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:42.170418 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dccab46-f31b-4a1b-ba97-d6fed4f58f16" path="/var/lib/kubelet/pods/9dccab46-f31b-4a1b-ba97-d6fed4f58f16/volumes" Jul 20 18:32:44.082876 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.082840 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm"] Jul 20 18:32:44.083301 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.083166 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5103e72-0afe-4f54-9280-f0d1adf301a2" containerName="istio-proxy" Jul 20 18:32:44.083301 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.083177 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5103e72-0afe-4f54-9280-f0d1adf301a2" containerName="istio-proxy" Jul 20 18:32:44.083301 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.083185 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dccab46-f31b-4a1b-ba97-d6fed4f58f16" containerName="istio-proxy" Jul 20 18:32:44.083301 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.083191 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dccab46-f31b-4a1b-ba97-d6fed4f58f16" containerName="istio-proxy" Jul 20 18:32:44.083301 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.083201 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5103e72-0afe-4f54-9280-f0d1adf301a2" containerName="istio-proxy" Jul 20 18:32:44.083301 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.083206 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5103e72-0afe-4f54-9280-f0d1adf301a2" containerName="istio-proxy" Jul 20 18:32:44.083301 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.083212 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dccab46-f31b-4a1b-ba97-d6fed4f58f16" containerName="istio-proxy" Jul 20 18:32:44.083301 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.083219 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dccab46-f31b-4a1b-ba97-d6fed4f58f16" containerName="istio-proxy" Jul 20 18:32:44.083301 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.083293 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dccab46-f31b-4a1b-ba97-d6fed4f58f16" containerName="istio-proxy" Jul 20 18:32:44.083301 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.083302 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5103e72-0afe-4f54-9280-f0d1adf301a2" containerName="istio-proxy" Jul 20 18:32:44.083611 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.083309 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dccab46-f31b-4a1b-ba97-d6fed4f58f16" containerName="istio-proxy" Jul 20 18:32:44.083611 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.083314 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5103e72-0afe-4f54-9280-f0d1adf301a2" containerName="istio-proxy" Jul 20 18:32:44.086128 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.086097 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.090123 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.090097 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-d56278-openshift-default-dockercfg-tbg8h\"" Jul 20 18:32:44.107546 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.107512 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm"] Jul 20 18:32:44.251596 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.251559 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-workload-socket\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.251596 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.251595 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-workload-certs\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.251857 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.251635 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6qdn\" (UniqueName: \"kubernetes.io/projected/00d42417-cbfc-4523-b04a-6630e7c1704f-kube-api-access-p6qdn\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.251857 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.251671 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-token\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.251857 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.251695 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-envoy\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.251857 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.251715 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-podinfo\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.251857 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.251791 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d42417-cbfc-4523-b04a-6630e7c1704f-istiod-ca-cert\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.251857 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.251827 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-data\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.251857 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.251855 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-credential-socket\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.353325 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.353240 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-data\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.353325 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.353283 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-credential-socket\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.353325 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.353313 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-workload-socket\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.353599 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.353336 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-workload-certs\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.353599 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.353361 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6qdn\" (UniqueName: \"kubernetes.io/projected/00d42417-cbfc-4523-b04a-6630e7c1704f-kube-api-access-p6qdn\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.353599 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.353384 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-token\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.353599 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.353416 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-envoy\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.353599 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.353438 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-podinfo\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.353599 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.353492 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d42417-cbfc-4523-b04a-6630e7c1704f-istiod-ca-cert\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.353961 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.353709 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-data\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.353961 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.353841 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-workload-certs\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.353961 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.353912 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-workload-socket\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.354115 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.353962 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-credential-socket\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.354187 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.354168 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d42417-cbfc-4523-b04a-6630e7c1704f-istiod-ca-cert\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.355857 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.355833 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-envoy\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.356096 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.356075 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-podinfo\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.362619 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.362594 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:32:44.364614 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.364587 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-token\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.365050 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.365029 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6qdn\" (UniqueName: \"kubernetes.io/projected/00d42417-cbfc-4523-b04a-6630e7c1704f-kube-api-access-p6qdn\") pod \"e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.397588 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.397546 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:44.549619 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.549592 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm"] Jul 20 18:32:44.552473 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:32:44.552442 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d42417_cbfc_4523_b04a_6630e7c1704f.slice/crio-ba71fa21ceac322a04eed9e8b73b6021790970b8a0d1f96fa2aa43bc3036ec47 WatchSource:0}: Error finding container ba71fa21ceac322a04eed9e8b73b6021790970b8a0d1f96fa2aa43bc3036ec47: Status 404 returned error can't find the container with id ba71fa21ceac322a04eed9e8b73b6021790970b8a0d1f96fa2aa43bc3036ec47 Jul 20 18:32:44.554474 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.554440 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:32:44.554553 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.554506 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:32:44.554553 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.554535 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:32:44.672396 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.672347 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" event={"ID":"00d42417-cbfc-4523-b04a-6630e7c1704f","Type":"ContainerStarted","Data":"cd98758f2a48015c368bbd6fb0738c24d7a8ad0aa3b6427cde36c2bf51b85323"} Jul 20 18:32:44.672597 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.672405 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" event={"ID":"00d42417-cbfc-4523-b04a-6630e7c1704f","Type":"ContainerStarted","Data":"ba71fa21ceac322a04eed9e8b73b6021790970b8a0d1f96fa2aa43bc3036ec47"} Jul 20 18:32:44.695682 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:44.695624 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" podStartSLOduration=0.695608987 podStartE2EDuration="695.608987ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:32:44.693265917 +0000 UTC m=+2673.065054495" watchObservedRunningTime="2026-07-20 18:32:44.695608987 +0000 UTC m=+2673.067397539" Jul 20 18:32:45.398644 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:45.398603 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:45.403704 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:45.403676 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:45.677953 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:45.677853 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:45.678738 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:45.678703 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:56.870206 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:56.870168 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:32:57.751534 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:57.751501 2583 generic.go:356] "Generic (PLEG): container finished" podID="00d42417-cbfc-4523-b04a-6630e7c1704f" containerID="cd98758f2a48015c368bbd6fb0738c24d7a8ad0aa3b6427cde36c2bf51b85323" exitCode=0 Jul 20 18:32:57.751786 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:57.751574 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" event={"ID":"00d42417-cbfc-4523-b04a-6630e7c1704f","Type":"ContainerDied","Data":"cd98758f2a48015c368bbd6fb0738c24d7a8ad0aa3b6427cde36c2bf51b85323"} Jul 20 18:32:57.751969 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:57.751954 2583 scope.go:117] "RemoveContainer" containerID="cd98758f2a48015c368bbd6fb0738c24d7a8ad0aa3b6427cde36c2bf51b85323" Jul 20 18:32:57.752999 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:57.752970 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:32:57.753072 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:57.753051 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:32:57.753114 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:57.753100 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 20 18:32:58.759194 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:58.759159 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" event={"ID":"00d42417-cbfc-4523-b04a-6630e7c1704f","Type":"ContainerStarted","Data":"3740e99d162a7e672ab9be4f7a55d301d1b00effc4906ddbc56c2ff81f99f5d8"} Jul 20 18:32:59.397741 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:59.397693 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:32:59.399041 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:59.399017 2583 patch_prober.go:28] interesting pod/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 20 18:32:59.399104 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:32:59.399066 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" podUID="00d42417-cbfc-4523-b04a-6630e7c1704f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 20 18:33:00.398233 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:00.398192 2583 patch_prober.go:28] interesting pod/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 20 18:33:00.398651 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:00.398262 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" podUID="00d42417-cbfc-4523-b04a-6630e7c1704f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 20 18:33:00.678372 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:00.678329 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:33:01.398574 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:01.398533 2583 patch_prober.go:28] interesting pod/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 20 18:33:01.399098 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:01.398611 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" podUID="00d42417-cbfc-4523-b04a-6630e7c1704f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 20 18:33:02.398118 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:02.398080 2583 patch_prober.go:28] interesting pod/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 20 18:33:02.398317 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:02.398141 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" podUID="00d42417-cbfc-4523-b04a-6630e7c1704f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 20 18:33:03.402391 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.402358 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:33:03.403526 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.403502 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:33:03.493709 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.493674 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w"] Jul 20 18:33:03.496274 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.496254 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.498893 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.498870 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-daa21b\"/\"auth-test-model-daa21b-kserve-self-signed-certs\"" Jul 20 18:33:03.499014 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.498905 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-daa21b\"/\"kube-root-ca.crt\"" Jul 20 18:33:03.499069 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.499009 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-daa21b\"/\"default-dockercfg-n5c6h\"" Jul 20 18:33:03.500116 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.500097 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-daa21b\"/\"openshift-service-ca.crt\"" Jul 20 18:33:03.509805 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.509777 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w"] Jul 20 18:33:03.529837 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.529800 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-model-cache\") pod \"auth-test-model-daa21b-kserve-5f7d6f947f-bf54w\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.529980 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.529838 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-home\") pod \"auth-test-model-daa21b-kserve-5f7d6f947f-bf54w\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.529980 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.529929 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9bb31bc-716a-410b-86d8-19d198f171f2-tls-certs\") pod \"auth-test-model-daa21b-kserve-5f7d6f947f-bf54w\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.530119 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.530006 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-tmp-dir\") pod \"auth-test-model-daa21b-kserve-5f7d6f947f-bf54w\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.530119 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.530036 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzm9k\" (UniqueName: \"kubernetes.io/projected/b9bb31bc-716a-410b-86d8-19d198f171f2-kube-api-access-pzm9k\") pod \"auth-test-model-daa21b-kserve-5f7d6f947f-bf54w\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.530119 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.530110 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-dshm\") pod \"auth-test-model-daa21b-kserve-5f7d6f947f-bf54w\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.631524 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.631476 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-dshm\") pod \"auth-test-model-daa21b-kserve-5f7d6f947f-bf54w\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.631524 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.631522 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-model-cache\") pod \"auth-test-model-daa21b-kserve-5f7d6f947f-bf54w\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.631843 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.631538 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-home\") pod \"auth-test-model-daa21b-kserve-5f7d6f947f-bf54w\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.631843 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.631582 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9bb31bc-716a-410b-86d8-19d198f171f2-tls-certs\") pod \"auth-test-model-daa21b-kserve-5f7d6f947f-bf54w\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.631843 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.631620 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-tmp-dir\") pod \"auth-test-model-daa21b-kserve-5f7d6f947f-bf54w\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.631843 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.631641 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzm9k\" (UniqueName: \"kubernetes.io/projected/b9bb31bc-716a-410b-86d8-19d198f171f2-kube-api-access-pzm9k\") pod \"auth-test-model-daa21b-kserve-5f7d6f947f-bf54w\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.632075 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.631998 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-model-cache\") pod \"auth-test-model-daa21b-kserve-5f7d6f947f-bf54w\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.632075 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.632019 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-home\") pod \"auth-test-model-daa21b-kserve-5f7d6f947f-bf54w\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.632146 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.632082 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-tmp-dir\") pod \"auth-test-model-daa21b-kserve-5f7d6f947f-bf54w\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.633902 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.633881 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-dshm\") pod \"auth-test-model-daa21b-kserve-5f7d6f947f-bf54w\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.634186 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.634168 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9bb31bc-716a-410b-86d8-19d198f171f2-tls-certs\") pod \"auth-test-model-daa21b-kserve-5f7d6f947f-bf54w\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.641475 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.641450 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzm9k\" (UniqueName: \"kubernetes.io/projected/b9bb31bc-716a-410b-86d8-19d198f171f2-kube-api-access-pzm9k\") pod \"auth-test-model-daa21b-kserve-5f7d6f947f-bf54w\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.806168 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.806126 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:03.941855 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:03.941830 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w"] Jul 20 18:33:03.943879 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:33:03.943849 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9bb31bc_716a_410b_86d8_19d198f171f2.slice/crio-ef04a2d4e779a016f8be365425ddc90e9c7e785e7296860863af68e50bf7c09f WatchSource:0}: Error finding container ef04a2d4e779a016f8be365425ddc90e9c7e785e7296860863af68e50bf7c09f: Status 404 returned error can't find the container with id ef04a2d4e779a016f8be365425ddc90e9c7e785e7296860863af68e50bf7c09f Jul 20 18:33:04.791153 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:04.791108 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" event={"ID":"b9bb31bc-716a-410b-86d8-19d198f171f2","Type":"ContainerStarted","Data":"6533125cae654f70860a98e1ff9413adf431bbe48a0386c2a9bad6be54829eb0"} Jul 20 18:33:04.791153 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:04.791165 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" event={"ID":"b9bb31bc-716a-410b-86d8-19d198f171f2","Type":"ContainerStarted","Data":"ef04a2d4e779a016f8be365425ddc90e9c7e785e7296860863af68e50bf7c09f"} Jul 20 18:33:04.811082 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:04.811016 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" podStartSLOduration=1.810995986 podStartE2EDuration="1.810995986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:33:04.810301095 +0000 UTC m=+2693.182089669" watchObservedRunningTime="2026-07-20 18:33:04.810995986 +0000 UTC m=+2693.182784560" Jul 20 18:33:09.453514 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:09.453480 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:33:12.234039 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:12.234012 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 18:33:12.246087 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:12.246060 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 18:33:13.806977 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:13.806941 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:13.807381 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:13.806994 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:13.819562 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:13.819530 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:13.846063 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:13.846033 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:33:46.586713 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:33:46.586676 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:34:03.257529 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:34:03.257492 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:34:22.350826 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:34:22.350793 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:34:36.650867 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:34:36.650784 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:34:55.768225 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:34:55.768189 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:35:11.552494 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:11.552459 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:35:25.358717 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:25.358681 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:35:42.956681 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:42.956641 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:35:46.518688 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:46.518655 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w"] Jul 20 18:35:49.696620 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.696577 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727"] Jul 20 18:35:49.699915 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.699896 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:49.702821 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.702797 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-daa21b\"/\"sub-test-model-daa21b-kserve-self-signed-certs\"" Jul 20 18:35:49.717122 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.717094 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727"] Jul 20 18:35:49.815140 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.815110 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8382eeec-5feb-46f4-84e9-f9ada9b0189c-tls-certs\") pod \"sub-test-model-daa21b-kserve-6bb7dd96c7-c2727\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:49.815291 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.815163 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndlrs\" (UniqueName: \"kubernetes.io/projected/8382eeec-5feb-46f4-84e9-f9ada9b0189c-kube-api-access-ndlrs\") pod \"sub-test-model-daa21b-kserve-6bb7dd96c7-c2727\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:49.815291 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.815224 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-home\") pod \"sub-test-model-daa21b-kserve-6bb7dd96c7-c2727\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:49.815291 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.815239 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-model-cache\") pod \"sub-test-model-daa21b-kserve-6bb7dd96c7-c2727\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:49.815291 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.815275 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-tmp-dir\") pod \"sub-test-model-daa21b-kserve-6bb7dd96c7-c2727\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:49.815437 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.815295 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-dshm\") pod \"sub-test-model-daa21b-kserve-6bb7dd96c7-c2727\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:49.916685 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.916643 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndlrs\" (UniqueName: \"kubernetes.io/projected/8382eeec-5feb-46f4-84e9-f9ada9b0189c-kube-api-access-ndlrs\") pod \"sub-test-model-daa21b-kserve-6bb7dd96c7-c2727\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:49.916885 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.916714 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-home\") pod \"sub-test-model-daa21b-kserve-6bb7dd96c7-c2727\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:49.916885 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.916760 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-model-cache\") pod \"sub-test-model-daa21b-kserve-6bb7dd96c7-c2727\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:49.916885 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.916784 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-tmp-dir\") pod \"sub-test-model-daa21b-kserve-6bb7dd96c7-c2727\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:49.916885 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.916802 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-dshm\") pod \"sub-test-model-daa21b-kserve-6bb7dd96c7-c2727\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:49.916885 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.916827 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8382eeec-5feb-46f4-84e9-f9ada9b0189c-tls-certs\") pod \"sub-test-model-daa21b-kserve-6bb7dd96c7-c2727\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:49.917144 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.917124 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-home\") pod \"sub-test-model-daa21b-kserve-6bb7dd96c7-c2727\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:49.917184 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.917170 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-model-cache\") pod \"sub-test-model-daa21b-kserve-6bb7dd96c7-c2727\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:49.917220 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.917202 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-tmp-dir\") pod \"sub-test-model-daa21b-kserve-6bb7dd96c7-c2727\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:49.919086 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.919065 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-dshm\") pod \"sub-test-model-daa21b-kserve-6bb7dd96c7-c2727\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:49.919335 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.919319 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8382eeec-5feb-46f4-84e9-f9ada9b0189c-tls-certs\") pod \"sub-test-model-daa21b-kserve-6bb7dd96c7-c2727\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:49.925756 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:49.925720 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndlrs\" (UniqueName: \"kubernetes.io/projected/8382eeec-5feb-46f4-84e9-f9ada9b0189c-kube-api-access-ndlrs\") pod \"sub-test-model-daa21b-kserve-6bb7dd96c7-c2727\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:50.009068 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:50.008994 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:35:50.140296 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:50.140264 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727"] Jul 20 18:35:50.143324 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:35:50.143297 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8382eeec_5feb_46f4_84e9_f9ada9b0189c.slice/crio-71880c903b7ef11d175e69b64de5936a8c78579598a73935f0cda7fced68ea65 WatchSource:0}: Error finding container 71880c903b7ef11d175e69b64de5936a8c78579598a73935f0cda7fced68ea65: Status 404 returned error can't find the container with id 71880c903b7ef11d175e69b64de5936a8c78579598a73935f0cda7fced68ea65 Jul 20 18:35:50.618671 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:50.618633 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" event={"ID":"8382eeec-5feb-46f4-84e9-f9ada9b0189c","Type":"ContainerStarted","Data":"c9892844d6c50341e2a9c265e7dff93e22ac275a5be05f8558723bc0b0850c65"} Jul 20 18:35:50.618866 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:50.618681 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" event={"ID":"8382eeec-5feb-46f4-84e9-f9ada9b0189c","Type":"ContainerStarted","Data":"71880c903b7ef11d175e69b64de5936a8c78579598a73935f0cda7fced68ea65"} Jul 20 18:35:50.640571 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:35:50.640524 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" podStartSLOduration=1.640510768 podStartE2EDuration="1.640510768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:35:50.638232682 +0000 UTC m=+2859.010021258" watchObservedRunningTime="2026-07-20 18:35:50.640510768 +0000 UTC m=+2859.012299342" Jul 20 18:36:00.009580 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:00.009545 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:36:00.010026 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:00.009598 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:36:00.021998 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:00.021972 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:36:00.680373 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:00.680344 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:36:01.538759 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:01.538687 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" podUID="b9bb31bc-716a-410b-86d8-19d198f171f2" containerName="main" containerID="cri-o://6533125cae654f70860a98e1ff9413adf431bbe48a0386c2a9bad6be54829eb0" gracePeriod=45 Jul 20 18:36:01.676320 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:01.676291 2583 generic.go:356] "Generic (PLEG): container finished" podID="b9bb31bc-716a-410b-86d8-19d198f171f2" containerID="6533125cae654f70860a98e1ff9413adf431bbe48a0386c2a9bad6be54829eb0" exitCode=0 Jul 20 18:36:01.676486 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:01.676368 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" event={"ID":"b9bb31bc-716a-410b-86d8-19d198f171f2","Type":"ContainerDied","Data":"6533125cae654f70860a98e1ff9413adf431bbe48a0386c2a9bad6be54829eb0"} Jul 20 18:36:01.789994 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:01.789935 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:36:01.853122 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:01.853089 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:36:01.922445 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:01.922409 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-model-cache\") pod \"b9bb31bc-716a-410b-86d8-19d198f171f2\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " Jul 20 18:36:01.922630 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:01.922492 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-dshm\") pod \"b9bb31bc-716a-410b-86d8-19d198f171f2\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " Jul 20 18:36:01.922630 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:01.922549 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9bb31bc-716a-410b-86d8-19d198f171f2-tls-certs\") pod \"b9bb31bc-716a-410b-86d8-19d198f171f2\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " Jul 20 18:36:01.922630 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:01.922568 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-home\") pod \"b9bb31bc-716a-410b-86d8-19d198f171f2\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " Jul 20 18:36:01.922630 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:01.922589 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-tmp-dir\") pod \"b9bb31bc-716a-410b-86d8-19d198f171f2\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " Jul 20 18:36:01.922630 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:01.922625 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzm9k\" (UniqueName: \"kubernetes.io/projected/b9bb31bc-716a-410b-86d8-19d198f171f2-kube-api-access-pzm9k\") pod \"b9bb31bc-716a-410b-86d8-19d198f171f2\" (UID: \"b9bb31bc-716a-410b-86d8-19d198f171f2\") " Jul 20 18:36:01.922942 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:01.922674 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "b9bb31bc-716a-410b-86d8-19d198f171f2" (UID: "b9bb31bc-716a-410b-86d8-19d198f171f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:36:01.922942 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:01.922853 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-home" (OuterVolumeSpecName: "home") pod "b9bb31bc-716a-410b-86d8-19d198f171f2" (UID: "b9bb31bc-716a-410b-86d8-19d198f171f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:36:01.922942 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:01.922861 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-model-cache\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:36:01.922942 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:01.922916 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b9bb31bc-716a-410b-86d8-19d198f171f2" (UID: "b9bb31bc-716a-410b-86d8-19d198f171f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:36:01.924853 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:01.924832 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-dshm" (OuterVolumeSpecName: "dshm") pod "b9bb31bc-716a-410b-86d8-19d198f171f2" (UID: "b9bb31bc-716a-410b-86d8-19d198f171f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:36:01.924950 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:01.924930 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9bb31bc-716a-410b-86d8-19d198f171f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9bb31bc-716a-410b-86d8-19d198f171f2" (UID: "b9bb31bc-716a-410b-86d8-19d198f171f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 18:36:01.925002 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:01.924946 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9bb31bc-716a-410b-86d8-19d198f171f2-kube-api-access-pzm9k" (OuterVolumeSpecName: "kube-api-access-pzm9k") pod "b9bb31bc-716a-410b-86d8-19d198f171f2" (UID: "b9bb31bc-716a-410b-86d8-19d198f171f2"). InnerVolumeSpecName "kube-api-access-pzm9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:36:02.023883 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:02.023842 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9bb31bc-716a-410b-86d8-19d198f171f2-tls-certs\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:36:02.023883 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:02.023872 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-home\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:36:02.023883 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:02.023881 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-tmp-dir\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:36:02.023883 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:02.023892 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzm9k\" (UniqueName: \"kubernetes.io/projected/b9bb31bc-716a-410b-86d8-19d198f171f2-kube-api-access-pzm9k\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:36:02.024232 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:02.023901 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9bb31bc-716a-410b-86d8-19d198f171f2-dshm\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:36:02.683074 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:02.682998 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" Jul 20 18:36:02.683074 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:02.683033 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w" event={"ID":"b9bb31bc-716a-410b-86d8-19d198f171f2","Type":"ContainerDied","Data":"ef04a2d4e779a016f8be365425ddc90e9c7e785e7296860863af68e50bf7c09f"} Jul 20 18:36:02.683575 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:02.683079 2583 scope.go:117] "RemoveContainer" containerID="6533125cae654f70860a98e1ff9413adf431bbe48a0386c2a9bad6be54829eb0" Jul 20 18:36:02.704387 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:02.704357 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w"] Jul 20 18:36:02.708648 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:02.708626 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-daa21b/auth-test-model-daa21b-kserve-5f7d6f947f-bf54w"] Jul 20 18:36:04.170252 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:04.170221 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9bb31bc-716a-410b-86d8-19d198f171f2" path="/var/lib/kubelet/pods/b9bb31bc-716a-410b-86d8-19d198f171f2/volumes" Jul 20 18:36:24.355153 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:24.355117 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:36:42.451436 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:36:42.451399 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:37:00.457196 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:00.457155 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:37:07.331693 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:07.331661 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727"] Jul 20 18:37:19.051912 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:19.051878 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:37:22.351216 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.351181 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" podUID="8382eeec-5feb-46f4-84e9-f9ada9b0189c" containerName="main" containerID="cri-o://c9892844d6c50341e2a9c265e7dff93e22ac275a5be05f8558723bc0b0850c65" gracePeriod=45 Jul 20 18:37:22.606808 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.606723 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:37:22.716057 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.716019 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-model-cache\") pod \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " Jul 20 18:37:22.716249 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.716073 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndlrs\" (UniqueName: \"kubernetes.io/projected/8382eeec-5feb-46f4-84e9-f9ada9b0189c-kube-api-access-ndlrs\") pod \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " Jul 20 18:37:22.716249 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.716098 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-dshm\") pod \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " Jul 20 18:37:22.716249 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.716115 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-home\") pod \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " Jul 20 18:37:22.716249 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.716157 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8382eeec-5feb-46f4-84e9-f9ada9b0189c-tls-certs\") pod \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " Jul 20 18:37:22.716249 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.716172 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-tmp-dir\") pod \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\" (UID: \"8382eeec-5feb-46f4-84e9-f9ada9b0189c\") " Jul 20 18:37:22.716523 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.716355 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-model-cache" (OuterVolumeSpecName: "model-cache") pod "8382eeec-5feb-46f4-84e9-f9ada9b0189c" (UID: "8382eeec-5feb-46f4-84e9-f9ada9b0189c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:37:22.716523 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.716370 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-home" (OuterVolumeSpecName: "home") pod "8382eeec-5feb-46f4-84e9-f9ada9b0189c" (UID: "8382eeec-5feb-46f4-84e9-f9ada9b0189c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:37:22.716523 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.716504 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8382eeec-5feb-46f4-84e9-f9ada9b0189c" (UID: "8382eeec-5feb-46f4-84e9-f9ada9b0189c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:37:22.718379 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.718341 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-dshm" (OuterVolumeSpecName: "dshm") pod "8382eeec-5feb-46f4-84e9-f9ada9b0189c" (UID: "8382eeec-5feb-46f4-84e9-f9ada9b0189c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:37:22.718504 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.718422 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8382eeec-5feb-46f4-84e9-f9ada9b0189c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8382eeec-5feb-46f4-84e9-f9ada9b0189c" (UID: "8382eeec-5feb-46f4-84e9-f9ada9b0189c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 18:37:22.718504 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.718458 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8382eeec-5feb-46f4-84e9-f9ada9b0189c-kube-api-access-ndlrs" (OuterVolumeSpecName: "kube-api-access-ndlrs") pod "8382eeec-5feb-46f4-84e9-f9ada9b0189c" (UID: "8382eeec-5feb-46f4-84e9-f9ada9b0189c"). InnerVolumeSpecName "kube-api-access-ndlrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:37:22.816787 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.816750 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-model-cache\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:37:22.816787 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.816783 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndlrs\" (UniqueName: \"kubernetes.io/projected/8382eeec-5feb-46f4-84e9-f9ada9b0189c-kube-api-access-ndlrs\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:37:22.817005 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.816799 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-dshm\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:37:22.817005 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.816812 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-home\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:37:22.817005 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.816824 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8382eeec-5feb-46f4-84e9-f9ada9b0189c-tls-certs\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:37:22.817005 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.816836 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8382eeec-5feb-46f4-84e9-f9ada9b0189c-tmp-dir\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:37:22.895242 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.895165 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg"] Jul 20 18:37:22.895507 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.895495 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8382eeec-5feb-46f4-84e9-f9ada9b0189c" containerName="main" Jul 20 18:37:22.895549 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.895508 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8382eeec-5feb-46f4-84e9-f9ada9b0189c" containerName="main" Jul 20 18:37:22.895549 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.895517 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9bb31bc-716a-410b-86d8-19d198f171f2" containerName="main" Jul 20 18:37:22.895549 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.895523 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9bb31bc-716a-410b-86d8-19d198f171f2" containerName="main" Jul 20 18:37:22.895651 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.895570 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9bb31bc-716a-410b-86d8-19d198f171f2" containerName="main" Jul 20 18:37:22.895651 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.895578 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8382eeec-5feb-46f4-84e9-f9ada9b0189c" containerName="main" Jul 20 18:37:22.898587 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.898568 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:22.905416 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.905394 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-daa21b\"/\"rate-test-model-daa21b-kserve-self-signed-certs\"" Jul 20 18:37:22.915905 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:22.915874 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg"] Jul 20 18:37:23.018793 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.018757 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309143a0-ebaf-42ca-aa56-207948e8b530-tls-certs\") pod \"rate-test-model-daa21b-kserve-55998c88ff-rrxvg\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.018793 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.018801 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrcw7\" (UniqueName: \"kubernetes.io/projected/309143a0-ebaf-42ca-aa56-207948e8b530-kube-api-access-jrcw7\") pod \"rate-test-model-daa21b-kserve-55998c88ff-rrxvg\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.019037 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.018819 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-dshm\") pod \"rate-test-model-daa21b-kserve-55998c88ff-rrxvg\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.019037 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.018846 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-home\") pod \"rate-test-model-daa21b-kserve-55998c88ff-rrxvg\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.019037 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.018907 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-tmp-dir\") pod \"rate-test-model-daa21b-kserve-55998c88ff-rrxvg\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.019037 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.019000 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-model-cache\") pod \"rate-test-model-daa21b-kserve-55998c88ff-rrxvg\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.093292 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.093259 2583 generic.go:356] "Generic (PLEG): container finished" podID="8382eeec-5feb-46f4-84e9-f9ada9b0189c" containerID="c9892844d6c50341e2a9c265e7dff93e22ac275a5be05f8558723bc0b0850c65" exitCode=0 Jul 20 18:37:23.093493 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.093299 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" Jul 20 18:37:23.093493 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.093321 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" event={"ID":"8382eeec-5feb-46f4-84e9-f9ada9b0189c","Type":"ContainerDied","Data":"c9892844d6c50341e2a9c265e7dff93e22ac275a5be05f8558723bc0b0850c65"} Jul 20 18:37:23.093493 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.093353 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727" event={"ID":"8382eeec-5feb-46f4-84e9-f9ada9b0189c","Type":"ContainerDied","Data":"71880c903b7ef11d175e69b64de5936a8c78579598a73935f0cda7fced68ea65"} Jul 20 18:37:23.093493 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.093370 2583 scope.go:117] "RemoveContainer" containerID="c9892844d6c50341e2a9c265e7dff93e22ac275a5be05f8558723bc0b0850c65" Jul 20 18:37:23.102038 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.102021 2583 scope.go:117] "RemoveContainer" containerID="c9892844d6c50341e2a9c265e7dff93e22ac275a5be05f8558723bc0b0850c65" Jul 20 18:37:23.102320 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:37:23.102303 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9892844d6c50341e2a9c265e7dff93e22ac275a5be05f8558723bc0b0850c65\": container with ID starting with c9892844d6c50341e2a9c265e7dff93e22ac275a5be05f8558723bc0b0850c65 not found: ID does not exist" containerID="c9892844d6c50341e2a9c265e7dff93e22ac275a5be05f8558723bc0b0850c65" Jul 20 18:37:23.102366 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.102330 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9892844d6c50341e2a9c265e7dff93e22ac275a5be05f8558723bc0b0850c65"} err="failed to get container status \"c9892844d6c50341e2a9c265e7dff93e22ac275a5be05f8558723bc0b0850c65\": rpc error: code = NotFound desc = could not find container \"c9892844d6c50341e2a9c265e7dff93e22ac275a5be05f8558723bc0b0850c65\": container with ID starting with c9892844d6c50341e2a9c265e7dff93e22ac275a5be05f8558723bc0b0850c65 not found: ID does not exist" Jul 20 18:37:23.116053 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.116024 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727"] Jul 20 18:37:23.119866 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.119840 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309143a0-ebaf-42ca-aa56-207948e8b530-tls-certs\") pod \"rate-test-model-daa21b-kserve-55998c88ff-rrxvg\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.120020 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.119891 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrcw7\" (UniqueName: \"kubernetes.io/projected/309143a0-ebaf-42ca-aa56-207948e8b530-kube-api-access-jrcw7\") pod \"rate-test-model-daa21b-kserve-55998c88ff-rrxvg\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.120020 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.119949 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-dshm\") pod \"rate-test-model-daa21b-kserve-55998c88ff-rrxvg\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.120020 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.120003 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-home\") pod \"rate-test-model-daa21b-kserve-55998c88ff-rrxvg\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.120203 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.120064 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-tmp-dir\") pod \"rate-test-model-daa21b-kserve-55998c88ff-rrxvg\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.120203 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.120098 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-model-cache\") pod \"rate-test-model-daa21b-kserve-55998c88ff-rrxvg\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.120575 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.120529 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-home\") pod \"rate-test-model-daa21b-kserve-55998c88ff-rrxvg\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.121133 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.121110 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-tmp-dir\") pod \"rate-test-model-daa21b-kserve-55998c88ff-rrxvg\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.121361 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.121342 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-model-cache\") pod \"rate-test-model-daa21b-kserve-55998c88ff-rrxvg\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.122478 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.122458 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-dshm\") pod \"rate-test-model-daa21b-kserve-55998c88ff-rrxvg\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.122571 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.122468 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-daa21b/sub-test-model-daa21b-kserve-6bb7dd96c7-c2727"] Jul 20 18:37:23.122795 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.122772 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309143a0-ebaf-42ca-aa56-207948e8b530-tls-certs\") pod \"rate-test-model-daa21b-kserve-55998c88ff-rrxvg\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.128029 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.128007 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrcw7\" (UniqueName: \"kubernetes.io/projected/309143a0-ebaf-42ca-aa56-207948e8b530-kube-api-access-jrcw7\") pod \"rate-test-model-daa21b-kserve-55998c88ff-rrxvg\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.207965 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.207926 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:23.346170 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:23.346066 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg"] Jul 20 18:37:23.349114 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:37:23.349084 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod309143a0_ebaf_42ca_aa56_207948e8b530.slice/crio-b50ebb25a899b7960124dee2e2757d6e536d5202b8a0752ac63df61435b318f0 WatchSource:0}: Error finding container b50ebb25a899b7960124dee2e2757d6e536d5202b8a0752ac63df61435b318f0: Status 404 returned error can't find the container with id b50ebb25a899b7960124dee2e2757d6e536d5202b8a0752ac63df61435b318f0 Jul 20 18:37:24.100285 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:24.100244 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" event={"ID":"309143a0-ebaf-42ca-aa56-207948e8b530","Type":"ContainerStarted","Data":"753e19b98bf110f7b3ff51da6236cec4f8b9bf3a4a20754901796d6ab800b65c"} Jul 20 18:37:24.100285 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:24.100292 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" event={"ID":"309143a0-ebaf-42ca-aa56-207948e8b530","Type":"ContainerStarted","Data":"b50ebb25a899b7960124dee2e2757d6e536d5202b8a0752ac63df61435b318f0"} Jul 20 18:37:24.122499 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:24.122439 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" podStartSLOduration=2.122422531 podStartE2EDuration="2.122422531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:37:24.120189388 +0000 UTC m=+2952.491977964" watchObservedRunningTime="2026-07-20 18:37:24.122422531 +0000 UTC m=+2952.494211104" Jul 20 18:37:24.170441 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:24.170402 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8382eeec-5feb-46f4-84e9-f9ada9b0189c" path="/var/lib/kubelet/pods/8382eeec-5feb-46f4-84e9-f9ada9b0189c/volumes" Jul 20 18:37:28.746493 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:28.746459 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:37:33.208825 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:33.208707 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:33.208825 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:33.208783 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:33.220933 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:33.220910 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:37:34.169469 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:37:34.169443 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:38:01.254908 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:38:01.254875 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:38:12.266629 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:38:12.266602 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 18:38:12.282583 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:38:12.282560 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 18:38:23.552173 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:38:23.552139 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:38:39.056283 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:38:39.056192 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:38:46.695356 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:38:46.695316 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg"] Jul 20 18:38:59.453438 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:38:59.453400 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:39:01.715286 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:01.715240 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" podUID="309143a0-ebaf-42ca-aa56-207948e8b530" containerName="main" containerID="cri-o://753e19b98bf110f7b3ff51da6236cec4f8b9bf3a4a20754901796d6ab800b65c" gracePeriod=45 Jul 20 18:39:01.964018 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:01.963993 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:39:02.137190 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.137104 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrcw7\" (UniqueName: \"kubernetes.io/projected/309143a0-ebaf-42ca-aa56-207948e8b530-kube-api-access-jrcw7\") pod \"309143a0-ebaf-42ca-aa56-207948e8b530\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " Jul 20 18:39:02.137190 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.137151 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309143a0-ebaf-42ca-aa56-207948e8b530-tls-certs\") pod \"309143a0-ebaf-42ca-aa56-207948e8b530\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " Jul 20 18:39:02.137190 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.137192 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-model-cache\") pod \"309143a0-ebaf-42ca-aa56-207948e8b530\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " Jul 20 18:39:02.137481 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.137218 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-tmp-dir\") pod \"309143a0-ebaf-42ca-aa56-207948e8b530\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " Jul 20 18:39:02.137481 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.137253 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-home\") pod \"309143a0-ebaf-42ca-aa56-207948e8b530\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " Jul 20 18:39:02.137481 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.137298 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-dshm\") pod \"309143a0-ebaf-42ca-aa56-207948e8b530\" (UID: \"309143a0-ebaf-42ca-aa56-207948e8b530\") " Jul 20 18:39:02.137635 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.137523 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-model-cache" (OuterVolumeSpecName: "model-cache") pod "309143a0-ebaf-42ca-aa56-207948e8b530" (UID: "309143a0-ebaf-42ca-aa56-207948e8b530"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:39:02.137635 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.137540 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "309143a0-ebaf-42ca-aa56-207948e8b530" (UID: "309143a0-ebaf-42ca-aa56-207948e8b530"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:39:02.137635 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.137553 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-home" (OuterVolumeSpecName: "home") pod "309143a0-ebaf-42ca-aa56-207948e8b530" (UID: "309143a0-ebaf-42ca-aa56-207948e8b530"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:39:02.139374 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.139351 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-dshm" (OuterVolumeSpecName: "dshm") pod "309143a0-ebaf-42ca-aa56-207948e8b530" (UID: "309143a0-ebaf-42ca-aa56-207948e8b530"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:39:02.139374 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.139350 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309143a0-ebaf-42ca-aa56-207948e8b530-kube-api-access-jrcw7" (OuterVolumeSpecName: "kube-api-access-jrcw7") pod "309143a0-ebaf-42ca-aa56-207948e8b530" (UID: "309143a0-ebaf-42ca-aa56-207948e8b530"). InnerVolumeSpecName "kube-api-access-jrcw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:39:02.139530 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.139414 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309143a0-ebaf-42ca-aa56-207948e8b530-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "309143a0-ebaf-42ca-aa56-207948e8b530" (UID: "309143a0-ebaf-42ca-aa56-207948e8b530"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 18:39:02.238272 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.238241 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrcw7\" (UniqueName: \"kubernetes.io/projected/309143a0-ebaf-42ca-aa56-207948e8b530-kube-api-access-jrcw7\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:39:02.238272 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.238266 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309143a0-ebaf-42ca-aa56-207948e8b530-tls-certs\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:39:02.238272 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.238278 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-model-cache\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:39:02.238473 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.238285 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-tmp-dir\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:39:02.238473 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.238292 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-home\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:39:02.238473 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.238299 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/309143a0-ebaf-42ca-aa56-207948e8b530-dshm\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:39:02.598580 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.598497 2583 generic.go:356] "Generic (PLEG): container finished" podID="309143a0-ebaf-42ca-aa56-207948e8b530" containerID="753e19b98bf110f7b3ff51da6236cec4f8b9bf3a4a20754901796d6ab800b65c" exitCode=0 Jul 20 18:39:02.598580 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.598533 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" Jul 20 18:39:02.598580 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.598550 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" event={"ID":"309143a0-ebaf-42ca-aa56-207948e8b530","Type":"ContainerDied","Data":"753e19b98bf110f7b3ff51da6236cec4f8b9bf3a4a20754901796d6ab800b65c"} Jul 20 18:39:02.598863 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.598589 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg" event={"ID":"309143a0-ebaf-42ca-aa56-207948e8b530","Type":"ContainerDied","Data":"b50ebb25a899b7960124dee2e2757d6e536d5202b8a0752ac63df61435b318f0"} Jul 20 18:39:02.598863 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.598607 2583 scope.go:117] "RemoveContainer" containerID="753e19b98bf110f7b3ff51da6236cec4f8b9bf3a4a20754901796d6ab800b65c" Jul 20 18:39:02.607117 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.607099 2583 scope.go:117] "RemoveContainer" containerID="753e19b98bf110f7b3ff51da6236cec4f8b9bf3a4a20754901796d6ab800b65c" Jul 20 18:39:02.607350 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:39:02.607328 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"753e19b98bf110f7b3ff51da6236cec4f8b9bf3a4a20754901796d6ab800b65c\": container with ID starting with 753e19b98bf110f7b3ff51da6236cec4f8b9bf3a4a20754901796d6ab800b65c not found: ID does not exist" containerID="753e19b98bf110f7b3ff51da6236cec4f8b9bf3a4a20754901796d6ab800b65c" Jul 20 18:39:02.607401 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.607360 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753e19b98bf110f7b3ff51da6236cec4f8b9bf3a4a20754901796d6ab800b65c"} err="failed to get container status \"753e19b98bf110f7b3ff51da6236cec4f8b9bf3a4a20754901796d6ab800b65c\": rpc error: code = NotFound desc = could not find container \"753e19b98bf110f7b3ff51da6236cec4f8b9bf3a4a20754901796d6ab800b65c\": container with ID starting with 753e19b98bf110f7b3ff51da6236cec4f8b9bf3a4a20754901796d6ab800b65c not found: ID does not exist" Jul 20 18:39:02.617143 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.617120 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg"] Jul 20 18:39:02.621386 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:02.621366 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-daa21b/rate-test-model-daa21b-kserve-55998c88ff-rrxvg"] Jul 20 18:39:04.173999 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:04.173964 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="309143a0-ebaf-42ca-aa56-207948e8b530" path="/var/lib/kubelet/pods/309143a0-ebaf-42ca-aa56-207948e8b530/volumes" Jul 20 18:39:20.554815 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:20.554773 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:39:42.464111 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:42.464067 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-h76fd"] Jul 20 18:39:51.482629 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:51.478852 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm"] Jul 20 18:39:51.482629 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:51.479196 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" podUID="00d42417-cbfc-4523-b04a-6630e7c1704f" containerName="istio-proxy" containerID="cri-o://3740e99d162a7e672ab9be4f7a55d301d1b00effc4906ddbc56c2ff81f99f5d8" gracePeriod=30 Jul 20 18:39:56.727074 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.727050 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:39:56.868275 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.868185 2583 generic.go:356] "Generic (PLEG): container finished" podID="00d42417-cbfc-4523-b04a-6630e7c1704f" containerID="3740e99d162a7e672ab9be4f7a55d301d1b00effc4906ddbc56c2ff81f99f5d8" exitCode=0 Jul 20 18:39:56.868275 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.868223 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" event={"ID":"00d42417-cbfc-4523-b04a-6630e7c1704f","Type":"ContainerDied","Data":"3740e99d162a7e672ab9be4f7a55d301d1b00effc4906ddbc56c2ff81f99f5d8"} Jul 20 18:39:56.868275 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.868230 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" Jul 20 18:39:56.868275 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.868276 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm" event={"ID":"00d42417-cbfc-4523-b04a-6630e7c1704f","Type":"ContainerDied","Data":"ba71fa21ceac322a04eed9e8b73b6021790970b8a0d1f96fa2aa43bc3036ec47"} Jul 20 18:39:56.868611 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.868304 2583 scope.go:117] "RemoveContainer" containerID="3740e99d162a7e672ab9be4f7a55d301d1b00effc4906ddbc56c2ff81f99f5d8" Jul 20 18:39:56.879843 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.879824 2583 scope.go:117] "RemoveContainer" containerID="cd98758f2a48015c368bbd6fb0738c24d7a8ad0aa3b6427cde36c2bf51b85323" Jul 20 18:39:56.879990 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.879970 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-workload-socket\") pod \"00d42417-cbfc-4523-b04a-6630e7c1704f\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " Jul 20 18:39:56.880097 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.879997 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-token\") pod \"00d42417-cbfc-4523-b04a-6630e7c1704f\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " Jul 20 18:39:56.880097 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.880023 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-data\") pod \"00d42417-cbfc-4523-b04a-6630e7c1704f\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " Jul 20 18:39:56.880097 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.880089 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6qdn\" (UniqueName: \"kubernetes.io/projected/00d42417-cbfc-4523-b04a-6630e7c1704f-kube-api-access-p6qdn\") pod \"00d42417-cbfc-4523-b04a-6630e7c1704f\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " Jul 20 18:39:56.880256 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.880119 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d42417-cbfc-4523-b04a-6630e7c1704f-istiod-ca-cert\") pod \"00d42417-cbfc-4523-b04a-6630e7c1704f\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " Jul 20 18:39:56.880256 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.880144 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-credential-socket\") pod \"00d42417-cbfc-4523-b04a-6630e7c1704f\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " Jul 20 18:39:56.880256 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.880182 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-podinfo\") pod \"00d42417-cbfc-4523-b04a-6630e7c1704f\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " Jul 20 18:39:56.880256 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.880218 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-envoy\") pod \"00d42417-cbfc-4523-b04a-6630e7c1704f\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " Jul 20 18:39:56.880256 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.880212 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "00d42417-cbfc-4523-b04a-6630e7c1704f" (UID: "00d42417-cbfc-4523-b04a-6630e7c1704f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:39:56.880518 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.880267 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-workload-certs\") pod \"00d42417-cbfc-4523-b04a-6630e7c1704f\" (UID: \"00d42417-cbfc-4523-b04a-6630e7c1704f\") " Jul 20 18:39:56.880518 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.880436 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "00d42417-cbfc-4523-b04a-6630e7c1704f" (UID: "00d42417-cbfc-4523-b04a-6630e7c1704f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:39:56.880634 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.880557 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-workload-socket\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:39:56.880634 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.880602 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-credential-socket\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:39:56.880634 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.880619 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "00d42417-cbfc-4523-b04a-6630e7c1704f" (UID: "00d42417-cbfc-4523-b04a-6630e7c1704f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:39:56.880959 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.880926 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d42417-cbfc-4523-b04a-6630e7c1704f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "00d42417-cbfc-4523-b04a-6630e7c1704f" (UID: "00d42417-cbfc-4523-b04a-6630e7c1704f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 18:39:56.881070 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.881044 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-data" (OuterVolumeSpecName: "istio-data") pod "00d42417-cbfc-4523-b04a-6630e7c1704f" (UID: "00d42417-cbfc-4523-b04a-6630e7c1704f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:39:56.882527 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.882494 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d42417-cbfc-4523-b04a-6630e7c1704f-kube-api-access-p6qdn" (OuterVolumeSpecName: "kube-api-access-p6qdn") pod "00d42417-cbfc-4523-b04a-6630e7c1704f" (UID: "00d42417-cbfc-4523-b04a-6630e7c1704f"). InnerVolumeSpecName "kube-api-access-p6qdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:39:56.882648 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.882628 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-token" (OuterVolumeSpecName: "istio-token") pod "00d42417-cbfc-4523-b04a-6630e7c1704f" (UID: "00d42417-cbfc-4523-b04a-6630e7c1704f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:39:56.882715 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.882661 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "00d42417-cbfc-4523-b04a-6630e7c1704f" (UID: "00d42417-cbfc-4523-b04a-6630e7c1704f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:39:56.882925 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.882904 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "00d42417-cbfc-4523-b04a-6630e7c1704f" (UID: "00d42417-cbfc-4523-b04a-6630e7c1704f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 18:39:56.898981 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.898956 2583 scope.go:117] "RemoveContainer" containerID="3740e99d162a7e672ab9be4f7a55d301d1b00effc4906ddbc56c2ff81f99f5d8" Jul 20 18:39:56.899255 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:39:56.899238 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3740e99d162a7e672ab9be4f7a55d301d1b00effc4906ddbc56c2ff81f99f5d8\": container with ID starting with 3740e99d162a7e672ab9be4f7a55d301d1b00effc4906ddbc56c2ff81f99f5d8 not found: ID does not exist" containerID="3740e99d162a7e672ab9be4f7a55d301d1b00effc4906ddbc56c2ff81f99f5d8" Jul 20 18:39:56.899343 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.899267 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3740e99d162a7e672ab9be4f7a55d301d1b00effc4906ddbc56c2ff81f99f5d8"} err="failed to get container status \"3740e99d162a7e672ab9be4f7a55d301d1b00effc4906ddbc56c2ff81f99f5d8\": rpc error: code = NotFound desc = could not find container \"3740e99d162a7e672ab9be4f7a55d301d1b00effc4906ddbc56c2ff81f99f5d8\": container with ID starting with 3740e99d162a7e672ab9be4f7a55d301d1b00effc4906ddbc56c2ff81f99f5d8 not found: ID does not exist" Jul 20 18:39:56.899343 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.899294 2583 scope.go:117] "RemoveContainer" containerID="cd98758f2a48015c368bbd6fb0738c24d7a8ad0aa3b6427cde36c2bf51b85323" Jul 20 18:39:56.899553 ip-10-0-141-129 kubenswrapper[2583]: E0720 18:39:56.899533 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd98758f2a48015c368bbd6fb0738c24d7a8ad0aa3b6427cde36c2bf51b85323\": container with ID starting with cd98758f2a48015c368bbd6fb0738c24d7a8ad0aa3b6427cde36c2bf51b85323 not found: ID does not exist" containerID="cd98758f2a48015c368bbd6fb0738c24d7a8ad0aa3b6427cde36c2bf51b85323" Jul 20 18:39:56.899600 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.899562 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd98758f2a48015c368bbd6fb0738c24d7a8ad0aa3b6427cde36c2bf51b85323"} err="failed to get container status \"cd98758f2a48015c368bbd6fb0738c24d7a8ad0aa3b6427cde36c2bf51b85323\": rpc error: code = NotFound desc = could not find container \"cd98758f2a48015c368bbd6fb0738c24d7a8ad0aa3b6427cde36c2bf51b85323\": container with ID starting with cd98758f2a48015c368bbd6fb0738c24d7a8ad0aa3b6427cde36c2bf51b85323 not found: ID does not exist" Jul 20 18:39:56.981950 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.981923 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-podinfo\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:39:56.981950 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.981945 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-envoy\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:39:56.981950 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.981955 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-workload-certs\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:39:56.982223 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.981962 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-token\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:39:56.982223 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.981970 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d42417-cbfc-4523-b04a-6630e7c1704f-istio-data\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:39:56.982223 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.981978 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6qdn\" (UniqueName: \"kubernetes.io/projected/00d42417-cbfc-4523-b04a-6630e7c1704f-kube-api-access-p6qdn\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:39:56.982223 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:56.981985 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d42417-cbfc-4523-b04a-6630e7c1704f-istiod-ca-cert\") on node \"ip-10-0-141-129.ec2.internal\" DevicePath \"\"" Jul 20 18:39:57.206081 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:57.206047 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm"] Jul 20 18:39:57.211373 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:57.211350 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-d56278-openshift-default-84cdb9cb5f-r79sm"] Jul 20 18:39:58.169883 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:58.169841 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00d42417-cbfc-4523-b04a-6630e7c1704f" path="/var/lib/kubelet/pods/00d42417-cbfc-4523-b04a-6630e7c1704f/volumes" Jul 20 18:39:59.298867 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:59.298844 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f8f8bd966-c4qlb_8eda8dbf-bc27-40e6-8f4a-698616904858/copy-manifests/0.log" Jul 20 18:39:59.318223 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:39:59.318197 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f8f8bd966-c4qlb_8eda8dbf-bc27-40e6-8f4a-698616904858/manager/0.log" Jul 20 18:40:01.247946 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:01.247913 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr_84002a57-b378-4606-8064-155b0b23bff1/util/0.log" Jul 20 18:40:01.258050 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:01.258019 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr_84002a57-b378-4606-8064-155b0b23bff1/pull/0.log" Jul 20 18:40:01.265670 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:01.265630 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr_84002a57-b378-4606-8064-155b0b23bff1/extract/0.log" Jul 20 18:40:01.386723 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:01.386689 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f_dbe65489-22b0-4198-b374-ed587a6a6d01/util/0.log" Jul 20 18:40:01.396855 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:01.396830 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f_dbe65489-22b0-4198-b374-ed587a6a6d01/pull/0.log" Jul 20 18:40:01.405190 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:01.405170 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f_dbe65489-22b0-4198-b374-ed587a6a6d01/extract/0.log" Jul 20 18:40:01.528590 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:01.528514 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn_0b183506-1a15-4994-905f-88d0ee42c349/util/0.log" Jul 20 18:40:01.536295 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:01.536270 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn_0b183506-1a15-4994-905f-88d0ee42c349/pull/0.log" Jul 20 18:40:01.543695 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:01.543678 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn_0b183506-1a15-4994-905f-88d0ee42c349/extract/0.log" Jul 20 18:40:01.672836 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:01.672801 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5_51dfed50-bc82-4dc1-ad65-ab84ced8504a/util/0.log" Jul 20 18:40:01.681264 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:01.681243 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5_51dfed50-bc82-4dc1-ad65-ab84ced8504a/pull/0.log" Jul 20 18:40:01.689289 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:01.689269 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5_51dfed50-bc82-4dc1-ad65-ab84ced8504a/extract/0.log" Jul 20 18:40:02.668260 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:02.668231 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-h76fd_e37cc553-2ea6-485e-ba1e-f4ca1001dd5e/limitador/0.log" Jul 20 18:40:03.371888 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:03.371856 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-z4njw_01389904-5991-4a81-9f66-7148e67defd0/discovery/0.log" Jul 20 18:40:04.691429 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:04.691403 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-wbr8q_3b85d5e8-dd54-4ef0-ac16-d0dd6cdd79d3/main/0.log" Jul 20 18:40:14.796818 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:14.796783 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rshx6_21af06f0-d59b-4e61-84fb-57425520c8c3/global-pull-secret-syncer/0.log" Jul 20 18:40:14.957641 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:14.957604 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v9j9l_7194a6f0-d226-438f-8445-2db548350c5b/konnectivity-agent/0.log" Jul 20 18:40:15.035123 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:15.035095 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-129.ec2.internal_7240dbae0531c45a7cd9f8a23f8cd206/haproxy/0.log" Jul 20 18:40:19.399276 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:19.399244 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr_84002a57-b378-4606-8064-155b0b23bff1/extract/0.log" Jul 20 18:40:19.426312 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:19.426285 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr_84002a57-b378-4606-8064-155b0b23bff1/util/0.log" Jul 20 18:40:19.453975 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:19.453950 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rzxpr_84002a57-b378-4606-8064-155b0b23bff1/pull/0.log" Jul 20 18:40:19.487384 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:19.487362 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f_dbe65489-22b0-4198-b374-ed587a6a6d01/extract/0.log" Jul 20 18:40:19.522384 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:19.522356 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f_dbe65489-22b0-4198-b374-ed587a6a6d01/util/0.log" Jul 20 18:40:19.554372 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:19.554351 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w6l9f_dbe65489-22b0-4198-b374-ed587a6a6d01/pull/0.log" Jul 20 18:40:19.607696 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:19.607663 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn_0b183506-1a15-4994-905f-88d0ee42c349/extract/0.log" Jul 20 18:40:19.643374 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:19.643346 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn_0b183506-1a15-4994-905f-88d0ee42c349/util/0.log" Jul 20 18:40:19.678322 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:19.678243 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p7fbn_0b183506-1a15-4994-905f-88d0ee42c349/pull/0.log" Jul 20 18:40:19.730019 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:19.729985 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5_51dfed50-bc82-4dc1-ad65-ab84ced8504a/extract/0.log" Jul 20 18:40:19.764083 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:19.764057 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5_51dfed50-bc82-4dc1-ad65-ab84ced8504a/util/0.log" Jul 20 18:40:19.796711 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:19.796684 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qw8p5_51dfed50-bc82-4dc1-ad65-ab84ced8504a/pull/0.log" Jul 20 18:40:20.778356 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:20.778325 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-h76fd_e37cc553-2ea6-485e-ba1e-f4ca1001dd5e/limitador/0.log" Jul 20 18:40:23.118346 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:23.118317 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hbfq6_d242fdc9-a9a1-4064-af05-7e2dee5d87c5/node-exporter/0.log" Jul 20 18:40:23.143266 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:23.143243 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hbfq6_d242fdc9-a9a1-4064-af05-7e2dee5d87c5/kube-rbac-proxy/0.log" Jul 20 18:40:23.174030 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:23.174008 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hbfq6_d242fdc9-a9a1-4064-af05-7e2dee5d87c5/init-textfile/0.log" Jul 20 18:40:23.390310 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:23.390221 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-65rhb_74fbff29-2608-4e88-a45f-dabff2fe558f/kube-rbac-proxy-main/0.log" Jul 20 18:40:23.415260 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:23.415235 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-65rhb_74fbff29-2608-4e88-a45f-dabff2fe558f/kube-rbac-proxy-self/0.log" Jul 20 18:40:23.438760 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:23.438721 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-65rhb_74fbff29-2608-4e88-a45f-dabff2fe558f/openshift-state-metrics/0.log" Jul 20 18:40:23.666702 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:23.666626 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-7nzqg_8a474b76-8147-4f5a-8bf6-933ca8943292/prometheus-operator/0.log" Jul 20 18:40:23.693360 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:23.693337 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-7nzqg_8a474b76-8147-4f5a-8bf6-933ca8943292/kube-rbac-proxy/0.log" Jul 20 18:40:25.115902 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.115870 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-pz8zz_f302722f-c610-42f7-972e-cb6885002372/networking-console-plugin/0.log" Jul 20 18:40:25.744320 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.744284 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj"] Jul 20 18:40:25.744607 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.744595 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d42417-cbfc-4523-b04a-6630e7c1704f" containerName="istio-proxy" Jul 20 18:40:25.744651 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.744608 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d42417-cbfc-4523-b04a-6630e7c1704f" containerName="istio-proxy" Jul 20 18:40:25.744651 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.744621 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d42417-cbfc-4523-b04a-6630e7c1704f" containerName="istio-proxy" Jul 20 18:40:25.744651 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.744627 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d42417-cbfc-4523-b04a-6630e7c1704f" containerName="istio-proxy" Jul 20 18:40:25.744651 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.744638 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="309143a0-ebaf-42ca-aa56-207948e8b530" containerName="main" Jul 20 18:40:25.744651 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.744645 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="309143a0-ebaf-42ca-aa56-207948e8b530" containerName="main" Jul 20 18:40:25.744838 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.744693 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="309143a0-ebaf-42ca-aa56-207948e8b530" containerName="main" Jul 20 18:40:25.744838 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.744701 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d42417-cbfc-4523-b04a-6630e7c1704f" containerName="istio-proxy" Jul 20 18:40:25.744838 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.744709 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d42417-cbfc-4523-b04a-6630e7c1704f" containerName="istio-proxy" Jul 20 18:40:25.747671 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.747654 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:25.751196 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.751178 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9gtvv\"/\"kube-root-ca.crt\"" Jul 20 18:40:25.752090 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.752068 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9gtvv\"/\"default-dockercfg-knh4w\"" Jul 20 18:40:25.752198 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.752115 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9gtvv\"/\"openshift-service-ca.crt\"" Jul 20 18:40:25.772834 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.772808 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj"] Jul 20 18:40:25.775385 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.775367 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/1.log" Jul 20 18:40:25.780511 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.780491 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-8v8sv_084f381e-6cd8-4547-9de6-9808e8655048/console-operator/2.log" Jul 20 18:40:25.816089 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.816059 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee910091-1168-49d3-8b00-9cead258dc32-sys\") pod \"perf-node-gather-daemonset-tcwqj\" (UID: \"ee910091-1168-49d3-8b00-9cead258dc32\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:25.816259 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.816100 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ee910091-1168-49d3-8b00-9cead258dc32-podres\") pod \"perf-node-gather-daemonset-tcwqj\" (UID: \"ee910091-1168-49d3-8b00-9cead258dc32\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:25.816259 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.816170 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee910091-1168-49d3-8b00-9cead258dc32-lib-modules\") pod \"perf-node-gather-daemonset-tcwqj\" (UID: \"ee910091-1168-49d3-8b00-9cead258dc32\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:25.816259 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.816197 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ee910091-1168-49d3-8b00-9cead258dc32-proc\") pod \"perf-node-gather-daemonset-tcwqj\" (UID: \"ee910091-1168-49d3-8b00-9cead258dc32\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:25.816259 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.816252 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khp4g\" (UniqueName: \"kubernetes.io/projected/ee910091-1168-49d3-8b00-9cead258dc32-kube-api-access-khp4g\") pod \"perf-node-gather-daemonset-tcwqj\" (UID: \"ee910091-1168-49d3-8b00-9cead258dc32\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:25.916847 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.916814 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khp4g\" (UniqueName: \"kubernetes.io/projected/ee910091-1168-49d3-8b00-9cead258dc32-kube-api-access-khp4g\") pod \"perf-node-gather-daemonset-tcwqj\" (UID: \"ee910091-1168-49d3-8b00-9cead258dc32\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:25.917004 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.916854 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee910091-1168-49d3-8b00-9cead258dc32-sys\") pod \"perf-node-gather-daemonset-tcwqj\" (UID: \"ee910091-1168-49d3-8b00-9cead258dc32\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:25.917004 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.916879 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ee910091-1168-49d3-8b00-9cead258dc32-podres\") pod \"perf-node-gather-daemonset-tcwqj\" (UID: \"ee910091-1168-49d3-8b00-9cead258dc32\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:25.917004 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.916917 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee910091-1168-49d3-8b00-9cead258dc32-lib-modules\") pod \"perf-node-gather-daemonset-tcwqj\" (UID: \"ee910091-1168-49d3-8b00-9cead258dc32\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:25.917004 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.916937 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ee910091-1168-49d3-8b00-9cead258dc32-proc\") pod \"perf-node-gather-daemonset-tcwqj\" (UID: \"ee910091-1168-49d3-8b00-9cead258dc32\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:25.917004 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.916956 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee910091-1168-49d3-8b00-9cead258dc32-sys\") pod \"perf-node-gather-daemonset-tcwqj\" (UID: \"ee910091-1168-49d3-8b00-9cead258dc32\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:25.917202 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.917021 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ee910091-1168-49d3-8b00-9cead258dc32-proc\") pod \"perf-node-gather-daemonset-tcwqj\" (UID: \"ee910091-1168-49d3-8b00-9cead258dc32\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:25.917202 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.917068 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ee910091-1168-49d3-8b00-9cead258dc32-podres\") pod \"perf-node-gather-daemonset-tcwqj\" (UID: \"ee910091-1168-49d3-8b00-9cead258dc32\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:25.917202 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.917128 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee910091-1168-49d3-8b00-9cead258dc32-lib-modules\") pod \"perf-node-gather-daemonset-tcwqj\" (UID: \"ee910091-1168-49d3-8b00-9cead258dc32\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:25.926885 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:25.926855 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khp4g\" (UniqueName: \"kubernetes.io/projected/ee910091-1168-49d3-8b00-9cead258dc32-kube-api-access-khp4g\") pod \"perf-node-gather-daemonset-tcwqj\" (UID: \"ee910091-1168-49d3-8b00-9cead258dc32\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:26.057084 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:26.056991 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:26.193002 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:26.192979 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj"] Jul 20 18:40:26.194524 ip-10-0-141-129 kubenswrapper[2583]: W0720 18:40:26.194493 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podee910091_1168_49d3_8b00_9cead258dc32.slice/crio-84b1bab1508f6d9bfa7ecf7fc4d7515f5ccbcd774b807cc6d6d5840b5a08dfb1 WatchSource:0}: Error finding container 84b1bab1508f6d9bfa7ecf7fc4d7515f5ccbcd774b807cc6d6d5840b5a08dfb1: Status 404 returned error can't find the container with id 84b1bab1508f6d9bfa7ecf7fc4d7515f5ccbcd774b807cc6d6d5840b5a08dfb1 Jul 20 18:40:27.015447 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:27.015409 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" event={"ID":"ee910091-1168-49d3-8b00-9cead258dc32","Type":"ContainerStarted","Data":"ac8078c40a296dd1887d8c54ab3c8c1abc32d2ab4ec40209ccb0c77ec775d042"} Jul 20 18:40:27.015664 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:27.015474 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:27.015664 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:27.015500 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" event={"ID":"ee910091-1168-49d3-8b00-9cead258dc32","Type":"ContainerStarted","Data":"84b1bab1508f6d9bfa7ecf7fc4d7515f5ccbcd774b807cc6d6d5840b5a08dfb1"} Jul 20 18:40:27.034501 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:27.034446 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" podStartSLOduration=2.034429633 podStartE2EDuration="2.034429633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 18:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:40:27.033030809 +0000 UTC m=+3135.404819382" watchObservedRunningTime="2026-07-20 18:40:27.034429633 +0000 UTC m=+3135.406218207" Jul 20 18:40:28.091539 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:28.091508 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qvw7j_9477e1ed-8e61-4f89-8cc3-db20326fce10/dns/0.log" Jul 20 18:40:28.123491 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:28.123467 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qvw7j_9477e1ed-8e61-4f89-8cc3-db20326fce10/kube-rbac-proxy/0.log" Jul 20 18:40:28.236809 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:28.236781 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qgk6s_167b1498-09ad-4ab3-84e9-8787235afb26/dns-node-resolver/0.log" Jul 20 18:40:28.815552 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:28.815521 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7cdfcc7df9-f7lqq_08c5249a-a8fe-47c5-816d-a8ed99cd2f44/registry/0.log" Jul 20 18:40:28.867541 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:28.867506 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7jrr6_0ff18797-615b-4cc4-82f2-3621080f562e/node-ca/0.log" Jul 20 18:40:30.005061 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:30.005023 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-z4njw_01389904-5991-4a81-9f66-7148e67defd0/discovery/0.log" Jul 20 18:40:30.958304 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:30.958274 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9rqgb_5c931ca0-fe6b-4e41-8bf8-53617767759f/serve-healthcheck-canary/0.log" Jul 20 18:40:31.590408 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:31.590384 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gkzdq_e8490308-78b2-45e7-b449-2126c3cd30a9/kube-rbac-proxy/0.log" Jul 20 18:40:31.617022 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:31.617001 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gkzdq_e8490308-78b2-45e7-b449-2126c3cd30a9/exporter/0.log" Jul 20 18:40:31.642931 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:31.642904 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gkzdq_e8490308-78b2-45e7-b449-2126c3cd30a9/extractor/0.log" Jul 20 18:40:33.030760 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:33.030675 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-tcwqj" Jul 20 18:40:34.351449 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:34.351411 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f8f8bd966-c4qlb_8eda8dbf-bc27-40e6-8f4a-698616904858/manager/0.log" Jul 20 18:40:34.370829 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:34.370803 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f8f8bd966-c4qlb_8eda8dbf-bc27-40e6-8f4a-698616904858/copy-manifests/0.log" Jul 20 18:40:34.864600 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:34.864559 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-z4njw_01389904-5991-4a81-9f66-7148e67defd0/discovery/0.log" Jul 20 18:40:35.939428 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:35.939401 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7974845875-rvrrz_cfb742af-fb9a-4d7f-b199-2d0f3d54fa11/manager/1.log" Jul 20 18:40:35.939955 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:35.939753 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7974845875-rvrrz_cfb742af-fb9a-4d7f-b199-2d0f3d54fa11/manager/0.log" Jul 20 18:40:36.461574 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:36.461536 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-z4njw_01389904-5991-4a81-9f66-7148e67defd0/discovery/0.log" Jul 20 18:40:41.034181 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:41.034147 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-bcrpq_0194534a-df67-4bad-a569-0e453aed46a2/migrator/0.log" Jul 20 18:40:41.056304 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:41.056277 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-bcrpq_0194534a-df67-4bad-a569-0e453aed46a2/graceful-termination/0.log" Jul 20 18:40:41.443615 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:41.443587 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-xrcw9_6f897e0b-72f4-40a1-917e-8e03196cedfe/kube-storage-version-migrator-operator/1.log" Jul 20 18:40:41.444475 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:41.444457 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-xrcw9_6f897e0b-72f4-40a1-917e-8e03196cedfe/kube-storage-version-migrator-operator/0.log" Jul 20 18:40:42.647646 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:42.647619 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2xsj4_e8583b5a-bbcd-44ba-9c03-2a14cfa40f19/kube-multus-additional-cni-plugins/0.log" Jul 20 18:40:42.686711 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:42.686682 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2xsj4_e8583b5a-bbcd-44ba-9c03-2a14cfa40f19/egress-router-binary-copy/0.log" Jul 20 18:40:42.723712 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:42.723691 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2xsj4_e8583b5a-bbcd-44ba-9c03-2a14cfa40f19/cni-plugins/0.log" Jul 20 18:40:42.764928 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:42.764896 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2xsj4_e8583b5a-bbcd-44ba-9c03-2a14cfa40f19/bond-cni-plugin/0.log" Jul 20 18:40:42.814645 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:42.814622 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2xsj4_e8583b5a-bbcd-44ba-9c03-2a14cfa40f19/routeoverride-cni/0.log" Jul 20 18:40:42.873607 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:42.873577 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2xsj4_e8583b5a-bbcd-44ba-9c03-2a14cfa40f19/whereabouts-cni-bincopy/0.log" Jul 20 18:40:42.921616 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:42.921534 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2xsj4_e8583b5a-bbcd-44ba-9c03-2a14cfa40f19/whereabouts-cni/0.log" Jul 20 18:40:43.336513 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:43.336480 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-q9g6b_a265dbdd-eaa3-465c-a32b-022b3b94f594/kube-multus/0.log" Jul 20 18:40:43.482172 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:43.482143 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r24gs_76af6cb3-7b9d-4993-a5c8-24afeb42b60b/network-metrics-daemon/0.log" Jul 20 18:40:43.525856 ip-10-0-141-129 kubenswrapper[2583]: I0720 18:40:43.525828 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r24gs_76af6cb3-7b9d-4993-a5c8-24afeb42b60b/kube-rbac-proxy/0.log"