Jul 10 16:36:20.461018 ip-10-0-134-41 systemd[1]: Starting Kubernetes Kubelet... Jul 10 16:36:20.924958 ip-10-0-134-41 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 16:36:20.924958 ip-10-0-134-41 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 16:36:20.924958 ip-10-0-134-41 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 16:36:20.924958 ip-10-0-134-41 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 16:36:20.924958 ip-10-0-134-41 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 16:36:20.924958 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.846579 2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 16:36:20.924958 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853480 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:36:20.924958 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853492 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:36:20.924958 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853495 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:36:20.924958 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853499 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:36:20.924958 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853502 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:36:20.924958 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853505 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:36:20.924958 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853509 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:36:20.924958 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853512 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:36:20.917275 ip-10-0-134-41 systemd[1]: Started Kubernetes Kubelet. Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853515 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853517 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853520 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853529 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853531 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853534 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853537 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853539 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853542 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853545 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853547 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853550 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853552 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853555 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853558 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853561 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853564 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853567 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853570 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:36:20.926335 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853573 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853576 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853580 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853584 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853588 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853591 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853593 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853597 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853600 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853602 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853606 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853608 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853611 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853613 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853616 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853619 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853621 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853624 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853628 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853631 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:36:20.927247 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853634 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853636 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853639 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853641 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853644 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853646 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853649 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853652 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853654 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853657 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853659 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853662 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853666 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853668 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853671 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853674 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853677 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853679 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853682 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853685 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:36:20.928070 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853687 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853690 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853692 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853695 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853697 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853700 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853706 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853709 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853712 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853715 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853718 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853721 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853724 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853727 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853730 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853733 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853735 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853738 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853754 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:36:20.928645 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853758 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853761 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853764 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853767 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853770 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853773 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853776 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853779 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853781 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853787 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853790 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853794 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.853798 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854469 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854477 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854479 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854482 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854485 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854487 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854490 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:36:20.929303 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854493 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854495 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854499 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854503 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854506 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854508 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854511 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854514 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854516 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854519 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854521 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854524 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854526 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854529 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854531 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854534 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854536 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854540 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854544 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:36:20.930175 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854547 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854550 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854555 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854559 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854561 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854564 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854567 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854570 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854574 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854576 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854579 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854582 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854584 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854587 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854589 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854592 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854595 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854597 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854600 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854603 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:36:20.931110 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854606 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854608 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854611 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854613 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854615 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854618 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854621 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854623 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854625 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854628 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854631 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854633 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854636 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854638 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854643 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854646 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854648 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854651 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854653 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854656 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:36:20.932762 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854659 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854661 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854663 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854667 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854669 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854671 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854674 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854677 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854679 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854682 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854684 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854687 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854690 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854693 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854696 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854698 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854701 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854703 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854706 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854708 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:36:20.933887 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854711 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854713 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854716 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854719 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854723 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854726 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854731 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854733 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854736 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854739 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854755 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854757 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.854760 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855106 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855111 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855118 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855121 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855123 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855126 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855128 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:36:20.934995 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855131 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855134 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855137 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855140 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855142 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855145 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855147 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855150 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855153 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855155 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855158 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855160 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855163 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855166 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855168 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855170 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855173 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855176 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855180 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855183 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:36:20.936008 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855185 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855188 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855190 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855193 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855195 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855197 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855200 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855203 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855206 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855208 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855213 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855216 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855220 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855223 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855225 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855228 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855231 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855234 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855237 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:36:20.936970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855239 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855242 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855244 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855247 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855249 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855252 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855254 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855257 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855259 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855262 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855264 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855268 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855270 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855273 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855275 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855278 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855280 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855282 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855285 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855288 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:36:20.939993 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855291 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855294 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855297 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855301 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855303 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855306 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855308 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855311 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855313 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855316 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855319 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855321 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855324 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855326 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855329 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855331 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855334 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855336 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855338 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:36:20.941044 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855341 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855343 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855346 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855350 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855353 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855356 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855358 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855361 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855365 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855368 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855371 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855374 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855377 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.855381 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855448 2585 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855458 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855464 2585 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855468 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855474 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855479 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855485 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 16:36:20.941928 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855491 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855494 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855498 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855502 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855505 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855508 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855511 2585 flags.go:64] FLAG: --cgroup-root="" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855514 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855517 2585 flags.go:64] FLAG: --client-ca-file="" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855520 2585 flags.go:64] FLAG: --cloud-provider="external" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855523 2585 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855527 2585 flags.go:64] FLAG: --cluster-domain="" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855530 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855533 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855536 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855540 2585 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855544 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855547 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855550 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855553 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855556 2585 flags.go:64] FLAG: --contention-profiling="false" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855559 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855562 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855565 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 16:36:20.943228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855569 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855573 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855576 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855578 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855581 2585 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855584 2585 flags.go:64] FLAG: --enable-server="true" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855587 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855594 2585 flags.go:64] FLAG: --event-burst="100" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855597 2585 flags.go:64] FLAG: --event-qps="50" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855600 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855603 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855606 2585 flags.go:64] FLAG: --eviction-hard="" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855610 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855614 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855616 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855619 2585 flags.go:64] FLAG: --eviction-soft="" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855623 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855626 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855628 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855631 2585 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855634 2585 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855637 2585 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855640 2585 flags.go:64] FLAG: --feature-gates="" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855644 2585 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855647 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 16:36:20.944285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855650 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855653 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855656 2585 flags.go:64] FLAG: --healthz-port="10248" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855659 2585 flags.go:64] FLAG: --help="false" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855662 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-134-41.ec2.internal" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855665 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855668 2585 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855672 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855675 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855678 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855681 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855684 2585 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855687 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855690 2585 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855693 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855696 2585 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855698 2585 flags.go:64] FLAG: --kube-reserved="" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855701 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855704 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855707 2585 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855710 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855713 2585 flags.go:64] FLAG: --lock-file="" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855716 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855718 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 16:36:20.945312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855721 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855727 2585 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855730 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855732 2585 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855735 2585 flags.go:64] FLAG: --logging-format="text" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855738 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855757 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855762 2585 flags.go:64] FLAG: --manifest-url="" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855767 2585 flags.go:64] FLAG: --manifest-url-header="" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855773 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855777 2585 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855781 2585 flags.go:64] FLAG: --max-pods="110" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855784 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855787 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855790 2585 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855795 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855798 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855801 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855804 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855810 2585 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855813 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855816 2585 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855819 2585 flags.go:64] FLAG: --pod-cidr="" Jul 10 16:36:20.946454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855822 2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855828 2585 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855831 2585 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855834 2585 flags.go:64] FLAG: --pods-per-core="0" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855837 2585 flags.go:64] FLAG: --port="10250" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855840 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855843 2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c51f21400efcd86e" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855846 2585 flags.go:64] FLAG: --qos-reserved="" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855849 2585 flags.go:64] FLAG: --read-only-port="10255" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855852 2585 flags.go:64] FLAG: --register-node="true" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855856 2585 flags.go:64] FLAG: --register-with-taints="" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855859 2585 flags.go:64] FLAG: --registry-burst="10" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855862 2585 flags.go:64] FLAG: --registry-qps="5" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855865 2585 flags.go:64] FLAG: --reserved-cpus="" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855867 2585 flags.go:64] FLAG: --reserved-memory="" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855871 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855875 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855878 2585 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855881 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855883 2585 flags.go:64] FLAG: --runonce="false" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855886 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855889 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855892 2585 flags.go:64] FLAG: --seccomp-default="false" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855895 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855899 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855902 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 16:36:20.950663 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855905 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855908 2585 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855911 2585 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855914 2585 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855916 2585 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855919 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855922 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855925 2585 flags.go:64] FLAG: --system-cgroups="" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855928 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855933 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855936 2585 flags.go:64] FLAG: --tls-cert-file="" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855939 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855943 2585 flags.go:64] FLAG: --tls-min-version="" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855946 2585 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855949 2585 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855951 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855954 2585 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855957 2585 flags.go:64] FLAG: --v="2" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855961 2585 flags.go:64] FLAG: --version="false" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855967 2585 flags.go:64] FLAG: --vmodule="" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855971 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.855974 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856104 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856108 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856111 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:36:20.952018 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856114 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856117 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856120 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856123 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856125 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856130 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856132 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856135 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856138 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856141 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856143 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856146 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856148 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856150 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856153 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856155 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856158 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856160 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856163 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856166 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:36:21.061240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856168 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856171 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856174 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856176 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856178 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856182 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856186 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856189 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856192 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856196 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856199 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856202 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856205 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856209 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856212 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856215 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856218 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856222 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856225 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:36:21.061991 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856227 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856230 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856232 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856235 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856237 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856240 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856243 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856245 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856248 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856251 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856253 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856256 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856258 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856261 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856264 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856267 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856269 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856272 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856274 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856277 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:36:21.062730 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856280 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856282 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856286 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856288 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856291 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856293 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856296 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856298 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856301 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856303 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856307 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856310 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856313 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856315 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856317 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856320 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856323 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856325 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856327 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:36:21.063470 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856330 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:36:21.064213 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856333 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:36:21.064213 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856336 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:36:21.064213 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856340 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:36:21.064213 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856343 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:36:21.064213 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856345 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:36:21.064213 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856348 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:36:21.064213 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856351 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:36:21.064213 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856353 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:36:21.064213 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856356 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:36:21.064213 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856359 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:36:21.064213 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856361 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:36:21.064213 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856364 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:36:21.064213 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856366 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:36:21.064213 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856369 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:36:21.064213 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856373 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:36:21.064213 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856376 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:36:21.064213 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.856378 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:36:21.064970 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.856942 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 16:36:21.064970 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.863677 2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 16:36:21.064970 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.863694 2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 16:36:21.064970 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.864603 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 16:36:21.064970 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.864611 2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 16:36:21.064970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864683 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:36:21.064970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864688 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:36:21.064970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864691 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:36:21.064970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864694 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:36:21.064970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864697 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:36:21.064970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864700 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:36:21.064970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864703 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:36:21.064970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864707 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:36:21.064970 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864712 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864715 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864718 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864721 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864723 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864726 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864729 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864731 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864734 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864736 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864739 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864754 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864757 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864760 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864762 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864765 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864768 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864770 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864773 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:36:21.065723 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864776 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864779 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864790 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864794 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864796 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864799 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864802 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864804 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864807 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864810 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864813 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864815 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864818 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864821 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864823 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864827 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864831 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864834 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864838 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864841 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:36:21.066480 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864844 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864848 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864850 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864853 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864856 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864858 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864861 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864863 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864866 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864869 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864871 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864877 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864879 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864882 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864885 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864894 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864898 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864901 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864904 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864907 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:36:21.067231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864910 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864913 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864915 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864918 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864921 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864924 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864927 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864929 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864932 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864935 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864937 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864940 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864942 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864945 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864947 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864950 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864952 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864955 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864957 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864960 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:36:21.067987 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864962 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:36:21.068727 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864965 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:36:21.068727 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864967 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:36:21.068727 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864971 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:36:21.068727 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864973 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:36:21.068727 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864976 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:36:21.068727 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864979 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:36:21.068727 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864981 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:36:21.068727 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864989 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:36:21.068727 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864992 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:36:21.068727 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864995 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:36:21.068727 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.864997 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:36:21.068727 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.865002 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 16:36:21.068727 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865153 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:36:21.068727 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865158 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:36:21.068727 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865161 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865165 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865168 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865171 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865174 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865177 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865179 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865182 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865185 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865187 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865190 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865193 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865196 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865198 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865201 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865203 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865206 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865208 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865211 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865213 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:36:21.069489 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865216 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865220 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865224 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865228 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865231 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865234 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865243 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865246 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865248 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865251 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865253 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865256 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865258 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865262 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865265 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865267 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865270 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865272 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865275 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865277 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:36:21.070222 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865280 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865283 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865285 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865288 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865290 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865293 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865295 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865298 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865300 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865303 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865305 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865308 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865311 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865314 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865317 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865319 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865322 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865324 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865327 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865337 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:36:21.070972 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865340 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865342 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865345 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865347 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865350 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865354 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865356 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865359 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865361 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865364 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865368 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865371 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865374 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865377 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865380 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865383 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865385 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865388 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865391 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:36:21.071702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865393 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:36:21.072450 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865396 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:36:21.072450 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865400 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:36:21.072450 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865403 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:36:21.072450 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865406 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:36:21.072450 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865409 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:36:21.072450 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865413 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:36:21.072450 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865416 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:36:21.072450 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865418 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:36:21.072450 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865421 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:36:21.072450 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865423 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:36:21.072450 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865426 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:36:21.072450 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865429 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:36:21.072450 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865437 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:36:21.072450 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865439 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:36:21.072450 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865442 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:36:21.072450 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865444 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:36:21.072450 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:20.865447 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:36:21.073192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.865453 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 16:36:21.073192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.865584 2585 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 16:36:21.073192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.867833 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 16:36:21.073192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.869177 2585 server.go:1019] "Starting client certificate rotation" Jul 10 16:36:21.073192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.869274 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 16:36:21.073192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.869309 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 16:36:21.073192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.871562 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 16:36:21.073192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.872912 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 16:36:21.073192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.886326 2585 log.go:25] "Validated CRI v1 runtime API" Jul 10 16:36:21.073192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.888329 2585 log.go:25] "Validated CRI v1 image API" Jul 10 16:36:21.073192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.889129 2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 16:36:21.073192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.892347 2585 fs.go:135] Filesystem UUIDs: map[43ebb889-fe00-4b03-a734-2cab41266fc9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cee48e98-9c54-4ddb-ab1b-cd9b77fbc218:/dev/nvme0n1p4] Jul 10 16:36:21.073583 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.892362 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 16:36:21.073960 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.898353 2585 manager.go:216] Machine: {Timestamp:2026-07-10 16:36:20.896981384 +0000 UTC m=+0.318857862 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3082376 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21d43238f09e494c8f6d51d0640543 SystemUUID:ec21d432-38f0-9e49-4c8f-6d51d0640543 BootID:dd293c32-fe2c-47ce-a109-2ac5302d9b94 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:84:7d:42:a7:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:84:7d:42:a7:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:20:7c:65:4b:9e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 16:36:21.073960 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.898446 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 16:36:21.073960 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.898534 2585 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 16:36:21.073960 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.899583 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 16:36:21.073960 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.902030 2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 16:36:21.073960 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.902058 2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-41.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 16:36:21.073960 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.902735 2585 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.902757 2585 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.902765 2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.903434 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.904507 2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.907649 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.907798 2585 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.907831 2585 kubelet.go:486] "Attempting to sync node with API server" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.907847 2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.907860 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.907870 2585 kubelet.go:398] "Adding apiserver pod source" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.907878 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.908166 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.908183 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.910793 2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.912014 2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.912060 2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.912692 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.912706 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.912714 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.912726 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.912737 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.912805 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.912814 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.912819 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.912825 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.912830 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.912839 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.912846 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.913591 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.913598 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.916583 2585 server.go:1268] "Started kubelet" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.916648 2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.916784 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.916870 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:20.917936 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-41.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.917992 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-41.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:20.918017 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.918133 2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 16:36:21.074320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.926980 2585 server.go:310] "Adding debug handlers to kubelet server" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:20.927651 2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.929215 2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.929243 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:20.929429 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.929845 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.929848 2585 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.929870 2585 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.929977 2585 reconstruct.go:97] "Volume reconstruction finished" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.929986 2585 reconciler.go:29] "Reconciler: start to sync state" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.930265 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.930278 2585 factory.go:55] Registering systemd factory Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.930287 2585 factory.go:223] Registration of the systemd container factory successfully Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.930466 2585 factory.go:153] Registering CRI-O factory Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.930478 2585 factory.go:223] Registration of the crio container factory successfully Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.930495 2585 factory.go:103] Registering Raw factory Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.930512 2585 manager.go:1196] Started watching for new ooms in manager Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.930881 2585 manager.go:349] Starting recovery of all containers Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:20.931715 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-41.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:20.932441 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:20.931786 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-41.ec2.internal.18c0faf49ae7b68f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-41.ec2.internal,UID:ip-10-0-134-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-41.ec2.internal,},FirstTimestamp:2026-07-10 16:36:20.916557455 +0000 UTC m=+0.338433913,LastTimestamp:2026-07-10 16:36:20.916557455 +0000 UTC m=+0.338433913,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-41.ec2.internal,}" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.937165 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sh6c8" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.944051 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sh6c8" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.957956 2585 manager.go:354] Recovery completed Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.961790 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.963620 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.963643 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.963669 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.964643 2585 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.964655 2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.964672 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.964817 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:36:21.078830 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.965673 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:36:21.174850 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.965701 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:36:21.174850 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.965712 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:36:21.174850 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.965897 2585 policy_none.go:49] "None policy: Start" Jul 10 16:36:21.174850 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.965915 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 16:36:21.174850 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.965930 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 16:36:21.174850 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:20.965976 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-41.ec2.internal.18c0faf49db5fdb6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-41.ec2.internal,UID:ip-10-0-134-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-41.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-41.ec2.internal,},FirstTimestamp:2026-07-10 16:36:20.963630518 +0000 UTC m=+0.385506978,LastTimestamp:2026-07-10 16:36:20.963630518 +0000 UTC m=+0.385506978,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-41.ec2.internal,}" Jul 10 16:36:21.174850 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:20.966982 2585 policy_none.go:47] "Start" Jul 10 16:36:21.326651 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:21.254655 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found" Jul 10 16:36:21.326651 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:21.260912 2585 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-41.ec2.internal\" not found" node="ip-10-0-134-41.ec2.internal" Jul 10 16:36:21.326651 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.289478 2585 manager.go:337] "Starting Device Plugin manager" Jul 10 16:36:21.326651 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:21.289511 2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 16:36:21.326651 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.289520 2585 server.go:87] "Starting device plugin registration server" Jul 10 16:36:21.326651 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.289710 2585 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 16:36:21.326651 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.289724 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 16:36:21.326651 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.291862 2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 16:36:21.326651 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.291968 2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 16:36:21.326651 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.291979 2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 16:36:21.326651 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:21.292123 2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 16:36:21.326651 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:21.292237 2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-41.ec2.internal\" not found" Jul 10 16:36:21.326651 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.317739 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 16:36:21.326651 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.318970 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 16:36:21.326651 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.319007 2585 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 16:36:21.326651 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.319027 2585 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 16:36:21.326651 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:21.319061 2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 16:36:21.326651 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.321853 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.390791 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.394683 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.394714 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.394729 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.394776 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-41.ec2.internal" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.403551 2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-41.ec2.internal" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:21.403569 2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-41.ec2.internal\": node \"ip-10-0-134-41.ec2.internal\" not found" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.419680 2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal"] Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.419759 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:21.420736 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.420842 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.420864 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.420874 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.422247 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.422400 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.422431 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.422886 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.422912 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.422927 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.422993 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.423012 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.423022 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.424196 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.424218 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.425037 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.425059 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:36:21.446030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.425075 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:36:21.458203 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.456622 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22c1e48ab6935889232428e887b9379d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal\" (UID: \"22c1e48ab6935889232428e887b9379d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" Jul 10 16:36:21.458203 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.456658 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e404ee0301f15458abcdaff3158e6ab9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-41.ec2.internal\" (UID: \"e404ee0301f15458abcdaff3158e6ab9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal" Jul 10 16:36:21.458203 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.456685 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22c1e48ab6935889232428e887b9379d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal\" (UID: \"22c1e48ab6935889232428e887b9379d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" Jul 10 16:36:21.560581 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:21.521598 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found" Jul 10 16:36:21.560581 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.556815 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22c1e48ab6935889232428e887b9379d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal\" (UID: \"22c1e48ab6935889232428e887b9379d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" Jul 10 16:36:21.560581 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.556848 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22c1e48ab6935889232428e887b9379d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal\" (UID: \"22c1e48ab6935889232428e887b9379d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" Jul 10 16:36:21.560581 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.556872 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e404ee0301f15458abcdaff3158e6ab9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-41.ec2.internal\" (UID: \"e404ee0301f15458abcdaff3158e6ab9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal" Jul 10 16:36:21.560581 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.556885 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22c1e48ab6935889232428e887b9379d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal\" (UID: \"22c1e48ab6935889232428e887b9379d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" Jul 10 16:36:21.560581 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.556921 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e404ee0301f15458abcdaff3158e6ab9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-41.ec2.internal\" (UID: \"e404ee0301f15458abcdaff3158e6ab9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal" Jul 10 16:36:21.560581 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.556965 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22c1e48ab6935889232428e887b9379d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal\" (UID: \"22c1e48ab6935889232428e887b9379d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" Jul 10 16:36:21.669378 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:21.622266 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found" Jul 10 16:36:21.777298 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:21.722916 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found" Jul 10 16:36:21.788142 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.786189 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:36:21.888394 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:21.823465 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found" Jul 10 16:36:21.888394 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.869193 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 16:36:21.888394 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.869328 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 16:36:21.888394 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.869349 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 16:36:22.002815 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:21.923533 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found" Jul 10 16:36:22.002815 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.955266 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 16:31:21 +0000 UTC" deadline="2027-12-13 02:52:41.778696411 +0000 UTC" Jul 10 16:36:22.002815 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:21.955285 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12490h16m19.823413993s" Jul 10 16:36:22.117082 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:22.023636 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found" Jul 10 16:36:22.117082 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.077096 2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:36:22.125485 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:22.124626 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found" Jul 10 16:36:22.235874 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:22.225266 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found" Jul 10 16:36:22.403747 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.290681 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:36:22.403747 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.295777 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" Jul 10 16:36:22.403747 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.299845 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal" Jul 10 16:36:22.403747 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.307835 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 16:36:22.403747 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.310456 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 16:36:22.403747 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.311317 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" Jul 10 16:36:22.403747 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.313054 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal" Jul 10 16:36:22.403747 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.329929 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal" Jul 10 16:36:22.403747 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.349159 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 16:36:22.403747 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:22.349205 2585 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-41.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal" Jul 10 16:36:22.403747 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.349220 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" Jul 10 16:36:22.403747 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.372523 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 16:36:22.403747 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:22.372556 2585 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" Jul 10 16:36:22.831662 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:22.831620 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c1e48ab6935889232428e887b9379d.slice/crio-e1249cbb4db9ce996d2af898f75272484bbb66ad01b6e01bd7fce9ce0f11d111 WatchSource:0}: Error finding container e1249cbb4db9ce996d2af898f75272484bbb66ad01b6e01bd7fce9ce0f11d111: Status 404 returned error can't find the container with id e1249cbb4db9ce996d2af898f75272484bbb66ad01b6e01bd7fce9ce0f11d111 Jul 10 16:36:22.836915 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.836883 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 16:36:22.910344 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.910320 2585 apiserver.go:52] "Watching apiserver" Jul 10 16:36:22.919333 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.919312 2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 16:36:22.920425 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.920403 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-nzk6t","kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal","openshift-cluster-node-tuning-operator/tuned-s7w2p","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal","openshift-multus/multus-additional-cni-plugins-fnqd8","openshift-network-diagnostics/network-check-target-6q44p","openshift-network-operator/iptables-alerter-dn8s4","openshift-ovn-kubernetes/ovnkube-node-4lqv6","kube-system/global-pull-secret-syncer-lhp9w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp","openshift-dns/node-resolver-m7vs8","openshift-image-registry/node-ca-dsv6h","openshift-multus/multus-fmt8m","openshift-multus/network-metrics-daemon-6c49f"] Jul 10 16:36:22.922838 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.922819 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:22.924135 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.924117 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fmt8m" Jul 10 16:36:22.925867 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.925850 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:22.925963 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.925871 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 16:36:22.926085 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.926064 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 16:36:22.927041 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.927018 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 16:36:22.927179 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.927159 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9d224\"" Jul 10 16:36:22.928797 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.927695 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:22.928797 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:22.927809 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:22.928797 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.927977 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 16:36:22.928797 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.928290 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 16:36:22.928797 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.928481 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 16:36:22.928797 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.928726 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6ggs4\"" Jul 10 16:36:22.930160 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.929221 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 16:36:22.930160 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.929596 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 16:36:22.930160 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.929610 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 16:36:22.930160 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.929742 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sclwt\"" Jul 10 16:36:22.930160 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.929876 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 16:36:22.931490 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.931467 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:22.932678 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.932660 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:22.932776 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:22.932721 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:22.934062 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.934042 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:22.934164 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.934087 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 16:36:22.934225 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:22.934203 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:22.934285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.934262 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 16:36:22.934338 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.934310 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s7zq2\"" Jul 10 16:36:22.935474 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.935459 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dn8s4" Jul 10 16:36:22.937260 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.937244 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:22.937991 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.937967 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 16:36:22.938076 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.937988 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c45jp\"" Jul 10 16:36:22.938442 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.938355 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 16:36:22.938442 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.938383 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 16:36:22.938763 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.938727 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nzk6t" Jul 10 16:36:22.940086 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.940067 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2xp4r\"" Jul 10 16:36:22.940181 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.940171 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 16:36:22.940245 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.940177 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 16:36:22.940245 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.940221 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 16:36:22.940345 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.940171 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 16:36:22.940345 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.940313 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m7vs8" Jul 10 16:36:22.940443 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.940394 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 16:36:22.941376 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.941358 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 16:36:22.941445 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.941393 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 16:36:22.941488 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.941446 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-g2jls\"" Jul 10 16:36:22.941780 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.941752 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dsv6h" Jul 10 16:36:22.942115 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.942017 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 16:36:22.942693 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.942679 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-flhsz\"" Jul 10 16:36:22.942863 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.942849 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 16:36:22.942933 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.942904 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 16:36:22.944042 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.944018 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 16:36:22.944407 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.944393 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 16:36:22.944407 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.944402 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bx8rx\"" Jul 10 16:36:22.944541 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.944528 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 16:36:22.944650 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.944633 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 16:36:22.963837 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.963817 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mgdft" Jul 10 16:36:22.967802 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.967785 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a75db21-5ca1-43b4-83f7-57bfe805b360-cnibin\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:22.967894 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.967812 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4a75db21-5ca1-43b4-83f7-57bfe805b360-os-release\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:22.967894 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.967857 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a75db21-5ca1-43b4-83f7-57bfe805b360-cni-binary-copy\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:22.967894 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.967885 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cf308df8-1c08-449a-91bf-f24e5485a3b4-registration-dir\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:22.967992 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.967905 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cf308df8-1c08-449a-91bf-f24e5485a3b4-device-dir\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:22.967992 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.967947 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-system-cni-dir\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:22.967992 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.967970 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-host-var-lib-cni-bin\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:22.967992 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.967989 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vgxm\" (UniqueName: \"kubernetes.io/projected/405fff59-3355-4cd2-9371-77c04c509283-kube-api-access-7vgxm\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:22.968129 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968021 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4a75db21-5ca1-43b4-83f7-57bfe805b360-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:22.968129 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968045 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml7rt\" (UniqueName: \"kubernetes.io/projected/cf308df8-1c08-449a-91bf-f24e5485a3b4-kube-api-access-ml7rt\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:22.968129 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968059 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-os-release\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:22.968129 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968077 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/405fff59-3355-4cd2-9371-77c04c509283-cni-binary-copy\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:22.968129 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968098 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-sysctl-conf\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:22.968301 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968132 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a75db21-5ca1-43b4-83f7-57bfe805b360-system-cni-dir\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:22.968301 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968160 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-host-run-netns\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:22.968301 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968181 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-multus-conf-dir\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:22.968301 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968197 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-modprobe-d\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:22.968301 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968217 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-sysconfig\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:22.968301 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968243 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-var-lib-kubelet\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:22.968301 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968261 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs\") pod \"network-metrics-daemon-6c49f\" (UID: \"08a4d929-2c36-40ac-a3e6-8c5caf0c1788\") " pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:22.968301 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968278 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cf308df8-1c08-449a-91bf-f24e5485a3b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:22.968301 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968298 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cf308df8-1c08-449a-91bf-f24e5485a3b4-socket-dir\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:22.968690 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968322 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cf308df8-1c08-449a-91bf-f24e5485a3b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:22.968690 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968343 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cf308df8-1c08-449a-91bf-f24e5485a3b4-sys-fs\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:22.968690 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968358 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-host-run-k8s-cni-cncf-io\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:22.968690 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968373 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-host-var-lib-kubelet\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:22.968690 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968409 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-kubernetes\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:22.968953 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968701 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4a75db21-5ca1-43b4-83f7-57bfe805b360-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:22.968953 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968758 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9cmk\" (UniqueName: \"kubernetes.io/projected/4a75db21-5ca1-43b4-83f7-57bfe805b360-kube-api-access-v9cmk\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:22.968953 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968815 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-cnibin\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:22.968953 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968841 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-multus-socket-dir-parent\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:22.968953 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968873 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-etc-kubernetes\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:22.968953 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968919 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-systemd\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:22.969177 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.968961 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-lib-modules\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:22.969177 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.969008 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhb5s\" (UniqueName: \"kubernetes.io/projected/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-kube-api-access-rhb5s\") pod \"network-metrics-daemon-6c49f\" (UID: \"08a4d929-2c36-40ac-a3e6-8c5caf0c1788\") " pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:22.969177 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.969063 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a75db21-5ca1-43b4-83f7-57bfe805b360-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:22.969177 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.969093 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-multus-cni-dir\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:22.969177 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.969137 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/405fff59-3355-4cd2-9371-77c04c509283-multus-daemon-config\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:22.969177 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.969171 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-hostroot\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:22.969479 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.969229 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-host-run-multus-certs\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:22.969479 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.969263 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-sysctl-d\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:22.969479 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.969299 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-run\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:22.969479 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.969328 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-host\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:22.969479 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.969352 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc77b858-26c5-406d-97ab-1dec3ec73678-tmp\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:22.969479 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.969383 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68xwn\" (UniqueName: \"kubernetes.io/projected/fc77b858-26c5-406d-97ab-1dec3ec73678-kube-api-access-68xwn\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:22.969479 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.969413 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-host-var-lib-cni-multus\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:22.969479 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.969444 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-sys\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:22.969940 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.969487 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-tuned\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:22.972693 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:22.972671 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mgdft" Jul 10 16:36:23.031469 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.031418 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 16:36:23.070243 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070223 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-run\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.070344 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070247 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-host\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.070344 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070264 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc77b858-26c5-406d-97ab-1dec3ec73678-tmp\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.070344 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070287 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-kubelet\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.070344 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070313 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-run-systemd\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.070344 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070338 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-etc-openvswitch\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.070585 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070344 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-host\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.070585 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070360 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-run-ovn\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.070585 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070404 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-host-var-lib-cni-multus\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.070585 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070434 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2594870a-c2d7-47e9-8f55-d6c029810a1d-dbus\") pod \"global-pull-secret-syncer-lhp9w\" (UID: \"2594870a-c2d7-47e9-8f55-d6c029810a1d\") " pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:23.070585 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070465 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-run\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.070585 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070483 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-host-var-lib-cni-multus\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.070585 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070527 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a75db21-5ca1-43b4-83f7-57bfe805b360-cnibin\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:23.070585 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070556 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a75db21-5ca1-43b4-83f7-57bfe805b360-cni-binary-copy\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:23.070585 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070579 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a75db21-5ca1-43b4-83f7-57bfe805b360-cnibin\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:23.070585 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070586 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbt2t\" (UniqueName: \"kubernetes.io/projected/1d54c0b4-7530-45fb-804c-777c15ff35eb-kube-api-access-lbt2t\") pod \"node-ca-dsv6h\" (UID: \"1d54c0b4-7530-45fb-804c-777c15ff35eb\") " pod="openshift-image-registry/node-ca-dsv6h" Jul 10 16:36:23.071142 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070571 2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 16:36:23.071142 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070617 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret\") pod \"global-pull-secret-syncer-lhp9w\" (UID: \"2594870a-c2d7-47e9-8f55-d6c029810a1d\") " pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:23.071142 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070642 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-systemd-units\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.071142 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070665 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-cni-netd\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.071142 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070692 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4a75db21-5ca1-43b4-83f7-57bfe805b360-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:23.071142 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070717 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml7rt\" (UniqueName: \"kubernetes.io/projected/cf308df8-1c08-449a-91bf-f24e5485a3b4-kube-api-access-ml7rt\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:23.071142 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070742 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-os-release\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.071142 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070786 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/405fff59-3355-4cd2-9371-77c04c509283-cni-binary-copy\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.071142 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070813 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2594870a-c2d7-47e9-8f55-d6c029810a1d-kubelet-config\") pod \"global-pull-secret-syncer-lhp9w\" (UID: \"2594870a-c2d7-47e9-8f55-d6c029810a1d\") " pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:23.071142 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070831 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-os-release\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.071142 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070838 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-host-run-netns\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.071142 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070883 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-modprobe-d\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.071142 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070908 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-sysconfig\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.071142 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070932 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-var-lib-kubelet\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.071815 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070955 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/696116b2-b4c2-458b-ad03-8b9f224ba24b-host-slash\") pod \"iptables-alerter-dn8s4\" (UID: \"696116b2-b4c2-458b-ad03-8b9f224ba24b\") " pod="openshift-network-operator/iptables-alerter-dn8s4" Jul 10 16:36:23.071815 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.070981 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph645\" (UniqueName: \"kubernetes.io/projected/696116b2-b4c2-458b-ad03-8b9f224ba24b-kube-api-access-ph645\") pod \"iptables-alerter-dn8s4\" (UID: \"696116b2-b4c2-458b-ad03-8b9f224ba24b\") " pod="openshift-network-operator/iptables-alerter-dn8s4" Jul 10 16:36:23.071815 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071005 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-run-netns\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.071815 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071049 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-node-log\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.071815 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071063 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-sysconfig\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.071815 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071072 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f99l\" (UniqueName: \"kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l\") pod \"network-check-target-6q44p\" (UID: \"a449c30f-b973-4076-b3da-484943344705\") " pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:23.071815 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071111 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cf308df8-1c08-449a-91bf-f24e5485a3b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:23.071815 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071121 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-host-run-netns\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.071815 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071165 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a75db21-5ca1-43b4-83f7-57bfe805b360-cni-binary-copy\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:23.071815 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071176 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-var-lib-kubelet\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.071815 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071212 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-modprobe-d\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.071815 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071216 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cf308df8-1c08-449a-91bf-f24e5485a3b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:23.071815 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071221 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-host-run-k8s-cni-cncf-io\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.071815 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071257 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-host-var-lib-kubelet\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.072524 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071264 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-host-run-k8s-cni-cncf-io\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.072524 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071292 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-host-var-lib-kubelet\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.072524 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071317 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4a75db21-5ca1-43b4-83f7-57bfe805b360-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:23.072524 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071322 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-ovnkube-config\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.072524 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071391 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4a75db21-5ca1-43b4-83f7-57bfe805b360-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:23.072524 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071432 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-cnibin\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.072524 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071459 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-multus-socket-dir-parent\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.072524 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071486 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-etc-kubernetes\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.072524 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071495 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-cnibin\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.072524 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071512 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhb5s\" (UniqueName: \"kubernetes.io/projected/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-kube-api-access-rhb5s\") pod \"network-metrics-daemon-6c49f\" (UID: \"08a4d929-2c36-40ac-a3e6-8c5caf0c1788\") " pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:23.072524 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071536 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-multus-socket-dir-parent\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.072524 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071539 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d54c0b4-7530-45fb-804c-777c15ff35eb-host\") pod \"node-ca-dsv6h\" (UID: \"1d54c0b4-7530-45fb-804c-777c15ff35eb\") " pod="openshift-image-registry/node-ca-dsv6h" Jul 10 16:36:23.072524 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071559 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-etc-kubernetes\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.072524 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071566 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d54c0b4-7530-45fb-804c-777c15ff35eb-serviceca\") pod \"node-ca-dsv6h\" (UID: \"1d54c0b4-7530-45fb-804c-777c15ff35eb\") " pod="openshift-image-registry/node-ca-dsv6h" Jul 10 16:36:23.073128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071592 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt7v2\" (UniqueName: \"kubernetes.io/projected/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-kube-api-access-gt7v2\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.073128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071636 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/405fff59-3355-4cd2-9371-77c04c509283-multus-daemon-config\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.073128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071662 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60be53b8-7bc0-456c-9029-76e0f65316e6-konnectivity-ca\") pod \"konnectivity-agent-nzk6t\" (UID: \"60be53b8-7bc0-456c-9029-76e0f65316e6\") " pod="kube-system/konnectivity-agent-nzk6t" Jul 10 16:36:23.073128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071702 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d492a8c-19e2-462a-85bc-ac78c1898d73-hosts-file\") pod \"node-resolver-m7vs8\" (UID: \"2d492a8c-19e2-462a-85bc-ac78c1898d73\") " pod="openshift-dns/node-resolver-m7vs8" Jul 10 16:36:23.073128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071804 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d492a8c-19e2-462a-85bc-ac78c1898d73-tmp-dir\") pod \"node-resolver-m7vs8\" (UID: \"2d492a8c-19e2-462a-85bc-ac78c1898d73\") " pod="openshift-dns/node-resolver-m7vs8" Jul 10 16:36:23.073128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071848 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-var-lib-openvswitch\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.073128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071875 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-log-socket\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.073128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071838 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4a75db21-5ca1-43b4-83f7-57bfe805b360-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:23.073128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071916 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-hostroot\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.073128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071946 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-host-run-multus-certs\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.073128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071976 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-sysctl-d\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.073128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.071993 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-host-run-multus-certs\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.073128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072002 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68xwn\" (UniqueName: \"kubernetes.io/projected/fc77b858-26c5-406d-97ab-1dec3ec73678-kube-api-access-68xwn\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.073128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072018 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-hostroot\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.073673 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072030 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t59b8\" (UniqueName: \"kubernetes.io/projected/2d492a8c-19e2-462a-85bc-ac78c1898d73-kube-api-access-t59b8\") pod \"node-resolver-m7vs8\" (UID: \"2d492a8c-19e2-462a-85bc-ac78c1898d73\") " pod="openshift-dns/node-resolver-m7vs8" Jul 10 16:36:23.073673 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072055 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.073673 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072067 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/405fff59-3355-4cd2-9371-77c04c509283-cni-binary-copy\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.073673 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072079 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-ovn-node-metrics-cert\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.073673 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072091 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-sysctl-d\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.073673 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072102 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-sys\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.073673 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072145 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-tuned\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.073673 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072174 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60be53b8-7bc0-456c-9029-76e0f65316e6-agent-certs\") pod \"konnectivity-agent-nzk6t\" (UID: \"60be53b8-7bc0-456c-9029-76e0f65316e6\") " pod="kube-system/konnectivity-agent-nzk6t" Jul 10 16:36:23.073673 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072171 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/405fff59-3355-4cd2-9371-77c04c509283-multus-daemon-config\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.073673 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072208 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-ovnkube-script-lib\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.073673 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072215 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-sys\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.073673 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072237 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4a75db21-5ca1-43b4-83f7-57bfe805b360-os-release\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:23.073673 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072263 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cf308df8-1c08-449a-91bf-f24e5485a3b4-registration-dir\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:23.073673 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072275 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4a75db21-5ca1-43b4-83f7-57bfe805b360-os-release\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:23.074192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072295 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cf308df8-1c08-449a-91bf-f24e5485a3b4-device-dir\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:23.074192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072310 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cf308df8-1c08-449a-91bf-f24e5485a3b4-registration-dir\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:23.074192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072321 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-system-cni-dir\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.074192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072346 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cf308df8-1c08-449a-91bf-f24e5485a3b4-device-dir\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:23.074192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072364 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-host-var-lib-cni-bin\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.074192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072390 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vgxm\" (UniqueName: \"kubernetes.io/projected/405fff59-3355-4cd2-9371-77c04c509283-kube-api-access-7vgxm\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.074192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072413 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-host-var-lib-cni-bin\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.074192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072391 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-system-cni-dir\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.074192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072415 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-sysctl-conf\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.074192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072457 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/696116b2-b4c2-458b-ad03-8b9f224ba24b-iptables-alerter-script\") pod \"iptables-alerter-dn8s4\" (UID: \"696116b2-b4c2-458b-ad03-8b9f224ba24b\") " pod="openshift-network-operator/iptables-alerter-dn8s4" Jul 10 16:36:23.074192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072486 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-run-openvswitch\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.074192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072524 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-sysctl-conf\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.074192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072536 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-run-ovn-kubernetes\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.074192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072563 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-env-overrides\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.074658 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072605 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a75db21-5ca1-43b4-83f7-57bfe805b360-system-cni-dir\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:23.074658 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072630 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-multus-conf-dir\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.074658 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072682 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a75db21-5ca1-43b4-83f7-57bfe805b360-system-cni-dir\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:23.074658 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072721 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs\") pod \"network-metrics-daemon-6c49f\" (UID: \"08a4d929-2c36-40ac-a3e6-8c5caf0c1788\") " pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:23.074658 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072768 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-cni-bin\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.074658 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072779 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-multus-conf-dir\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.074658 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072796 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-lib-modules\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.074658 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072825 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cf308df8-1c08-449a-91bf-f24e5485a3b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:23.074658 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072850 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cf308df8-1c08-449a-91bf-f24e5485a3b4-socket-dir\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:23.074658 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:23.072856 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:23.074658 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072873 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cf308df8-1c08-449a-91bf-f24e5485a3b4-sys-fs\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:23.074658 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072934 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-lib-modules\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.074658 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072966 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cf308df8-1c08-449a-91bf-f24e5485a3b4-sys-fs\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:23.074658 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072899 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-kubernetes\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.074658 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.072919 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cf308df8-1c08-449a-91bf-f24e5485a3b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:23.075179 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:23.073032 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs podName:08a4d929-2c36-40ac-a3e6-8c5caf0c1788 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:23.572997603 +0000 UTC m=+2.994874050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs") pod "network-metrics-daemon-6c49f" (UID: "08a4d929-2c36-40ac-a3e6-8c5caf0c1788") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:23.075179 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.073017 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cf308df8-1c08-449a-91bf-f24e5485a3b4-socket-dir\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:23.075179 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.073059 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-kubernetes\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.075179 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.073101 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9cmk\" (UniqueName: \"kubernetes.io/projected/4a75db21-5ca1-43b4-83f7-57bfe805b360-kube-api-access-v9cmk\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:23.075179 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.073131 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-systemd\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.075179 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.073173 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a75db21-5ca1-43b4-83f7-57bfe805b360-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:23.075179 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.073204 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-multus-cni-dir\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.075179 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.073231 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-systemd\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.075179 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.073256 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-slash\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.075179 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.073341 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a75db21-5ca1-43b4-83f7-57bfe805b360-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:23.075179 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.073358 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/405fff59-3355-4cd2-9371-77c04c509283-multus-cni-dir\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.075179 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.074235 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc77b858-26c5-406d-97ab-1dec3ec73678-tmp\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.075179 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.074467 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc77b858-26c5-406d-97ab-1dec3ec73678-etc-tuned\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.079919 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.079893 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml7rt\" (UniqueName: \"kubernetes.io/projected/cf308df8-1c08-449a-91bf-f24e5485a3b4-kube-api-access-ml7rt\") pod \"aws-ebs-csi-driver-node-rtzhp\" (UID: \"cf308df8-1c08-449a-91bf-f24e5485a3b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:23.080021 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.079982 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68xwn\" (UniqueName: \"kubernetes.io/projected/fc77b858-26c5-406d-97ab-1dec3ec73678-kube-api-access-68xwn\") pod \"tuned-s7w2p\" (UID: \"fc77b858-26c5-406d-97ab-1dec3ec73678\") " pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.080257 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.080237 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhb5s\" (UniqueName: \"kubernetes.io/projected/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-kube-api-access-rhb5s\") pod \"network-metrics-daemon-6c49f\" (UID: \"08a4d929-2c36-40ac-a3e6-8c5caf0c1788\") " pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:23.081486 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.081464 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9cmk\" (UniqueName: \"kubernetes.io/projected/4a75db21-5ca1-43b4-83f7-57bfe805b360-kube-api-access-v9cmk\") pod \"multus-additional-cni-plugins-fnqd8\" (UID: \"4a75db21-5ca1-43b4-83f7-57bfe805b360\") " pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:23.082254 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.082236 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vgxm\" (UniqueName: \"kubernetes.io/projected/405fff59-3355-4cd2-9371-77c04c509283-kube-api-access-7vgxm\") pod \"multus-fmt8m\" (UID: \"405fff59-3355-4cd2-9371-77c04c509283\") " pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.173876 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.173846 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-ovnkube-config\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.173987 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.173895 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d54c0b4-7530-45fb-804c-777c15ff35eb-host\") pod \"node-ca-dsv6h\" (UID: \"1d54c0b4-7530-45fb-804c-777c15ff35eb\") " pod="openshift-image-registry/node-ca-dsv6h" Jul 10 16:36:23.173987 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.173928 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d54c0b4-7530-45fb-804c-777c15ff35eb-serviceca\") pod \"node-ca-dsv6h\" (UID: \"1d54c0b4-7530-45fb-804c-777c15ff35eb\") " pod="openshift-image-registry/node-ca-dsv6h" Jul 10 16:36:23.173987 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.173946 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt7v2\" (UniqueName: \"kubernetes.io/projected/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-kube-api-access-gt7v2\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.173987 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.173966 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d54c0b4-7530-45fb-804c-777c15ff35eb-host\") pod \"node-ca-dsv6h\" (UID: \"1d54c0b4-7530-45fb-804c-777c15ff35eb\") " pod="openshift-image-registry/node-ca-dsv6h" Jul 10 16:36:23.174173 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.173970 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60be53b8-7bc0-456c-9029-76e0f65316e6-konnectivity-ca\") pod \"konnectivity-agent-nzk6t\" (UID: \"60be53b8-7bc0-456c-9029-76e0f65316e6\") " pod="kube-system/konnectivity-agent-nzk6t" Jul 10 16:36:23.174173 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174012 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d492a8c-19e2-462a-85bc-ac78c1898d73-hosts-file\") pod \"node-resolver-m7vs8\" (UID: \"2d492a8c-19e2-462a-85bc-ac78c1898d73\") " pod="openshift-dns/node-resolver-m7vs8" Jul 10 16:36:23.174173 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174030 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d492a8c-19e2-462a-85bc-ac78c1898d73-tmp-dir\") pod \"node-resolver-m7vs8\" (UID: \"2d492a8c-19e2-462a-85bc-ac78c1898d73\") " pod="openshift-dns/node-resolver-m7vs8" Jul 10 16:36:23.174173 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174051 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-var-lib-openvswitch\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.174173 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174080 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-log-socket\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.174173 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174109 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t59b8\" (UniqueName: \"kubernetes.io/projected/2d492a8c-19e2-462a-85bc-ac78c1898d73-kube-api-access-t59b8\") pod \"node-resolver-m7vs8\" (UID: \"2d492a8c-19e2-462a-85bc-ac78c1898d73\") " pod="openshift-dns/node-resolver-m7vs8" Jul 10 16:36:23.174173 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174110 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d492a8c-19e2-462a-85bc-ac78c1898d73-hosts-file\") pod \"node-resolver-m7vs8\" (UID: \"2d492a8c-19e2-462a-85bc-ac78c1898d73\") " pod="openshift-dns/node-resolver-m7vs8" Jul 10 16:36:23.174173 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174134 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.174173 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174162 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-log-socket\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.174173 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174162 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-ovn-node-metrics-cert\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.174653 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174202 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60be53b8-7bc0-456c-9029-76e0f65316e6-agent-certs\") pod \"konnectivity-agent-nzk6t\" (UID: \"60be53b8-7bc0-456c-9029-76e0f65316e6\") " pod="kube-system/konnectivity-agent-nzk6t" Jul 10 16:36:23.174653 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174232 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-var-lib-openvswitch\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.174653 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174242 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-ovnkube-script-lib\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.174653 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174287 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/696116b2-b4c2-458b-ad03-8b9f224ba24b-iptables-alerter-script\") pod \"iptables-alerter-dn8s4\" (UID: \"696116b2-b4c2-458b-ad03-8b9f224ba24b\") " pod="openshift-network-operator/iptables-alerter-dn8s4" Jul 10 16:36:23.174653 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174317 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-run-openvswitch\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.174653 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174342 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-run-ovn-kubernetes\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.174653 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174366 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-env-overrides\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.174653 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174402 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-cni-bin\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.174653 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174437 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-slash\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.174653 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174471 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-kubelet\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.174653 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174470 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d54c0b4-7530-45fb-804c-777c15ff35eb-serviceca\") pod \"node-ca-dsv6h\" (UID: \"1d54c0b4-7530-45fb-804c-777c15ff35eb\") " pod="openshift-image-registry/node-ca-dsv6h" Jul 10 16:36:23.174653 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174498 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-run-systemd\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.174653 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174522 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-etc-openvswitch\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.174653 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174548 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-run-ovn\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.174653 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174563 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-ovnkube-config\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.175364 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174574 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2594870a-c2d7-47e9-8f55-d6c029810a1d-dbus\") pod \"global-pull-secret-syncer-lhp9w\" (UID: \"2594870a-c2d7-47e9-8f55-d6c029810a1d\") " pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:23.175364 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174600 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60be53b8-7bc0-456c-9029-76e0f65316e6-konnectivity-ca\") pod \"konnectivity-agent-nzk6t\" (UID: \"60be53b8-7bc0-456c-9029-76e0f65316e6\") " pod="kube-system/konnectivity-agent-nzk6t" Jul 10 16:36:23.175364 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174615 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbt2t\" (UniqueName: \"kubernetes.io/projected/1d54c0b4-7530-45fb-804c-777c15ff35eb-kube-api-access-lbt2t\") pod \"node-ca-dsv6h\" (UID: \"1d54c0b4-7530-45fb-804c-777c15ff35eb\") " pod="openshift-image-registry/node-ca-dsv6h" Jul 10 16:36:23.175364 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174629 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2594870a-c2d7-47e9-8f55-d6c029810a1d-dbus\") pod \"global-pull-secret-syncer-lhp9w\" (UID: \"2594870a-c2d7-47e9-8f55-d6c029810a1d\") " pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:23.175364 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174646 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret\") pod \"global-pull-secret-syncer-lhp9w\" (UID: \"2594870a-c2d7-47e9-8f55-d6c029810a1d\") " pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:23.175364 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174671 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-systemd-units\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.175364 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174695 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-cni-netd\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.175364 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174727 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2594870a-c2d7-47e9-8f55-d6c029810a1d-kubelet-config\") pod \"global-pull-secret-syncer-lhp9w\" (UID: \"2594870a-c2d7-47e9-8f55-d6c029810a1d\") " pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:23.175364 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174757 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/696116b2-b4c2-458b-ad03-8b9f224ba24b-host-slash\") pod \"iptables-alerter-dn8s4\" (UID: \"696116b2-b4c2-458b-ad03-8b9f224ba24b\") " pod="openshift-network-operator/iptables-alerter-dn8s4" Jul 10 16:36:23.175364 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174750 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.175364 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174783 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph645\" (UniqueName: \"kubernetes.io/projected/696116b2-b4c2-458b-ad03-8b9f224ba24b-kube-api-access-ph645\") pod \"iptables-alerter-dn8s4\" (UID: \"696116b2-b4c2-458b-ad03-8b9f224ba24b\") " pod="openshift-network-operator/iptables-alerter-dn8s4" Jul 10 16:36:23.175364 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174785 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-run-openvswitch\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.175364 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174806 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-systemd-units\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.175364 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174809 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-run-netns\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.176119 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174860 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-node-log\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.176119 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174875 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2594870a-c2d7-47e9-8f55-d6c029810a1d-kubelet-config\") pod \"global-pull-secret-syncer-lhp9w\" (UID: \"2594870a-c2d7-47e9-8f55-d6c029810a1d\") " pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:23.176119 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174885 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f99l\" (UniqueName: \"kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l\") pod \"network-check-target-6q44p\" (UID: \"a449c30f-b973-4076-b3da-484943344705\") " pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:23.176119 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174919 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-cni-netd\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.176119 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:23.175008 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:23.176119 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.175013 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-ovnkube-script-lib\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.176119 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.175022 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-env-overrides\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.176119 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.175045 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/696116b2-b4c2-458b-ad03-8b9f224ba24b-host-slash\") pod \"iptables-alerter-dn8s4\" (UID: \"696116b2-b4c2-458b-ad03-8b9f224ba24b\") " pod="openshift-network-operator/iptables-alerter-dn8s4" Jul 10 16:36:23.176119 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:23.175074 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret podName:2594870a-c2d7-47e9-8f55-d6c029810a1d nodeName:}" failed. No retries permitted until 2026-07-10 16:36:23.675054813 +0000 UTC m=+3.096931260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret") pod "global-pull-secret-syncer-lhp9w" (UID: "2594870a-c2d7-47e9-8f55-d6c029810a1d") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:23.176119 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.175076 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-cni-bin\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.176119 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.175091 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-run-netns\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.176119 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.174466 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d492a8c-19e2-462a-85bc-ac78c1898d73-tmp-dir\") pod \"node-resolver-m7vs8\" (UID: \"2d492a8c-19e2-462a-85bc-ac78c1898d73\") " pod="openshift-dns/node-resolver-m7vs8" Jul 10 16:36:23.176119 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.175140 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-run-systemd\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.176119 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.175148 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-kubelet\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.176853 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.175143 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-slash\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.176853 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.175101 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-node-log\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.176853 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.175181 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-etc-openvswitch\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.176853 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.175186 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/696116b2-b4c2-458b-ad03-8b9f224ba24b-iptables-alerter-script\") pod \"iptables-alerter-dn8s4\" (UID: \"696116b2-b4c2-458b-ad03-8b9f224ba24b\") " pod="openshift-network-operator/iptables-alerter-dn8s4" Jul 10 16:36:23.176853 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.175193 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-run-ovn\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.176853 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.175213 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-host-run-ovn-kubernetes\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.176853 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.176456 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60be53b8-7bc0-456c-9029-76e0f65316e6-agent-certs\") pod \"konnectivity-agent-nzk6t\" (UID: \"60be53b8-7bc0-456c-9029-76e0f65316e6\") " pod="kube-system/konnectivity-agent-nzk6t" Jul 10 16:36:23.177297 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.177278 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-ovn-node-metrics-cert\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.184264 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:23.184244 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:36:23.184264 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:23.184263 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:36:23.184408 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:23.184273 2585 projected.go:196] Error preparing data for projected volume kube-api-access-8f99l for pod openshift-network-diagnostics/network-check-target-6q44p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:23.184408 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:23.184327 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l podName:a449c30f-b973-4076-b3da-484943344705 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:23.684313445 +0000 UTC m=+3.106189897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8f99l" (UniqueName: "kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l") pod "network-check-target-6q44p" (UID: "a449c30f-b973-4076-b3da-484943344705") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:23.185326 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.185305 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt7v2\" (UniqueName: \"kubernetes.io/projected/b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56-kube-api-access-gt7v2\") pod \"ovnkube-node-4lqv6\" (UID: \"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.186148 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.186128 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t59b8\" (UniqueName: \"kubernetes.io/projected/2d492a8c-19e2-462a-85bc-ac78c1898d73-kube-api-access-t59b8\") pod \"node-resolver-m7vs8\" (UID: \"2d492a8c-19e2-462a-85bc-ac78c1898d73\") " pod="openshift-dns/node-resolver-m7vs8" Jul 10 16:36:23.187296 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.187279 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbt2t\" (UniqueName: \"kubernetes.io/projected/1d54c0b4-7530-45fb-804c-777c15ff35eb-kube-api-access-lbt2t\") pod \"node-ca-dsv6h\" (UID: \"1d54c0b4-7530-45fb-804c-777c15ff35eb\") " pod="openshift-image-registry/node-ca-dsv6h" Jul 10 16:36:23.187789 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.187771 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph645\" (UniqueName: \"kubernetes.io/projected/696116b2-b4c2-458b-ad03-8b9f224ba24b-kube-api-access-ph645\") pod \"iptables-alerter-dn8s4\" (UID: \"696116b2-b4c2-458b-ad03-8b9f224ba24b\") " pod="openshift-network-operator/iptables-alerter-dn8s4" Jul 10 16:36:23.235496 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.235473 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" Jul 10 16:36:23.241015 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.240997 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fmt8m" Jul 10 16:36:23.243796 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:23.243772 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf308df8_1c08_449a_91bf_f24e5485a3b4.slice/crio-f6a921c80b21e09d75504df538e89ac9f5f5825fa12fe963537ee7aa69a624cb WatchSource:0}: Error finding container f6a921c80b21e09d75504df538e89ac9f5f5825fa12fe963537ee7aa69a624cb: Status 404 returned error can't find the container with id f6a921c80b21e09d75504df538e89ac9f5f5825fa12fe963537ee7aa69a624cb Jul 10 16:36:23.248420 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:23.248397 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405fff59_3355_4cd2_9371_77c04c509283.slice/crio-f26c9ef473d90ffc006856ce44513ceeca901c25c1f6b0744a131a5ba54d210c WatchSource:0}: Error finding container f26c9ef473d90ffc006856ce44513ceeca901c25c1f6b0744a131a5ba54d210c: Status 404 returned error can't find the container with id f26c9ef473d90ffc006856ce44513ceeca901c25c1f6b0744a131a5ba54d210c Jul 10 16:36:23.249279 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.249259 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" Jul 10 16:36:23.254886 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.254864 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dsv6h" Jul 10 16:36:23.258246 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:23.258224 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc77b858_26c5_406d_97ab_1dec3ec73678.slice/crio-a3f5efdc58a6b183e262804b2a3a3b86245b721cde328f3b11897692f92f9a8b WatchSource:0}: Error finding container a3f5efdc58a6b183e262804b2a3a3b86245b721cde328f3b11897692f92f9a8b: Status 404 returned error can't find the container with id a3f5efdc58a6b183e262804b2a3a3b86245b721cde328f3b11897692f92f9a8b Jul 10 16:36:23.259750 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.259728 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fnqd8" Jul 10 16:36:23.266789 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:23.266767 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d54c0b4_7530_45fb_804c_777c15ff35eb.slice/crio-19e840f9adf91f5c5a5da6ef1bca2fee389ef99f6baf7978ceb67a2d7de2f938 WatchSource:0}: Error finding container 19e840f9adf91f5c5a5da6ef1bca2fee389ef99f6baf7978ceb67a2d7de2f938: Status 404 returned error can't find the container with id 19e840f9adf91f5c5a5da6ef1bca2fee389ef99f6baf7978ceb67a2d7de2f938 Jul 10 16:36:23.268996 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.268979 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dn8s4" Jul 10 16:36:23.275975 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.275952 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:23.276358 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:23.276337 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a75db21_5ca1_43b4_83f7_57bfe805b360.slice/crio-87aab08108247aaf103e4288a153302ef854d93c64a0f7b44b4955c9566c72d8 WatchSource:0}: Error finding container 87aab08108247aaf103e4288a153302ef854d93c64a0f7b44b4955c9566c72d8: Status 404 returned error can't find the container with id 87aab08108247aaf103e4288a153302ef854d93c64a0f7b44b4955c9566c72d8 Jul 10 16:36:23.277637 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:23.277598 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode404ee0301f15458abcdaff3158e6ab9.slice/crio-1ef53ffe56eeb973883fc44a3b31010ef3e0670993bd0e027d1be13ceae568b1 WatchSource:0}: Error finding container 1ef53ffe56eeb973883fc44a3b31010ef3e0670993bd0e027d1be13ceae568b1: Status 404 returned error can't find the container with id 1ef53ffe56eeb973883fc44a3b31010ef3e0670993bd0e027d1be13ceae568b1 Jul 10 16:36:23.279774 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:23.279726 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696116b2_b4c2_458b_ad03_8b9f224ba24b.slice/crio-c5bf98fbf9ab99cec85b5f378b3418ddd3117c4ad7ff89553af539270e82574d WatchSource:0}: Error finding container c5bf98fbf9ab99cec85b5f378b3418ddd3117c4ad7ff89553af539270e82574d: Status 404 returned error can't find the container with id c5bf98fbf9ab99cec85b5f378b3418ddd3117c4ad7ff89553af539270e82574d Jul 10 16:36:23.282101 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.282009 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nzk6t" Jul 10 16:36:23.282191 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.282159 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m7vs8" Jul 10 16:36:23.286702 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:23.286680 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e0a636_8e9b_42fb_bfb6_b4bc543a4a56.slice/crio-03eefc297e44e5cd97173319d6490e86b41e3822fdc5a8900e71775366f61ff6 WatchSource:0}: Error finding container 03eefc297e44e5cd97173319d6490e86b41e3822fdc5a8900e71775366f61ff6: Status 404 returned error can't find the container with id 03eefc297e44e5cd97173319d6490e86b41e3822fdc5a8900e71775366f61ff6 Jul 10 16:36:23.293883 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:23.293835 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60be53b8_7bc0_456c_9029_76e0f65316e6.slice/crio-d61f650565a317a11ff76fbfa764c20b0cd0ba7c9777916dccdf189f5b55935d WatchSource:0}: Error finding container d61f650565a317a11ff76fbfa764c20b0cd0ba7c9777916dccdf189f5b55935d: Status 404 returned error can't find the container with id d61f650565a317a11ff76fbfa764c20b0cd0ba7c9777916dccdf189f5b55935d Jul 10 16:36:23.294889 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:23.294852 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d492a8c_19e2_462a_85bc_ac78c1898d73.slice/crio-4a2ae5f81bf7ad78cf71f7e6ae1d51b24e01b73ce7e7bbf77b168436b73938f5 WatchSource:0}: Error finding container 4a2ae5f81bf7ad78cf71f7e6ae1d51b24e01b73ce7e7bbf77b168436b73938f5: Status 404 returned error can't find the container with id 4a2ae5f81bf7ad78cf71f7e6ae1d51b24e01b73ce7e7bbf77b168436b73938f5 Jul 10 16:36:23.324768 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.324708 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" event={"ID":"fc77b858-26c5-406d-97ab-1dec3ec73678","Type":"ContainerStarted","Data":"a3f5efdc58a6b183e262804b2a3a3b86245b721cde328f3b11897692f92f9a8b"} Jul 10 16:36:23.326120 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.326089 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" event={"ID":"22c1e48ab6935889232428e887b9379d","Type":"ContainerStarted","Data":"e1249cbb4db9ce996d2af898f75272484bbb66ad01b6e01bd7fce9ce0f11d111"} Jul 10 16:36:23.327349 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.327325 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" event={"ID":"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56","Type":"ContainerStarted","Data":"03eefc297e44e5cd97173319d6490e86b41e3822fdc5a8900e71775366f61ff6"} Jul 10 16:36:23.328638 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.328611 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dn8s4" event={"ID":"696116b2-b4c2-458b-ad03-8b9f224ba24b","Type":"ContainerStarted","Data":"c5bf98fbf9ab99cec85b5f378b3418ddd3117c4ad7ff89553af539270e82574d"} Jul 10 16:36:23.329912 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.329885 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal" event={"ID":"e404ee0301f15458abcdaff3158e6ab9","Type":"ContainerStarted","Data":"1ef53ffe56eeb973883fc44a3b31010ef3e0670993bd0e027d1be13ceae568b1"} Jul 10 16:36:23.331208 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.331181 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dsv6h" event={"ID":"1d54c0b4-7530-45fb-804c-777c15ff35eb","Type":"ContainerStarted","Data":"19e840f9adf91f5c5a5da6ef1bca2fee389ef99f6baf7978ceb67a2d7de2f938"} Jul 10 16:36:23.332545 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.332524 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fmt8m" event={"ID":"405fff59-3355-4cd2-9371-77c04c509283","Type":"ContainerStarted","Data":"f26c9ef473d90ffc006856ce44513ceeca901c25c1f6b0744a131a5ba54d210c"} Jul 10 16:36:23.333708 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.333686 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" event={"ID":"cf308df8-1c08-449a-91bf-f24e5485a3b4","Type":"ContainerStarted","Data":"f6a921c80b21e09d75504df538e89ac9f5f5825fa12fe963537ee7aa69a624cb"} Jul 10 16:36:23.335027 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.335000 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m7vs8" event={"ID":"2d492a8c-19e2-462a-85bc-ac78c1898d73","Type":"ContainerStarted","Data":"4a2ae5f81bf7ad78cf71f7e6ae1d51b24e01b73ce7e7bbf77b168436b73938f5"} Jul 10 16:36:23.336249 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.336223 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nzk6t" event={"ID":"60be53b8-7bc0-456c-9029-76e0f65316e6","Type":"ContainerStarted","Data":"d61f650565a317a11ff76fbfa764c20b0cd0ba7c9777916dccdf189f5b55935d"} Jul 10 16:36:23.337675 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.337654 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fnqd8" event={"ID":"4a75db21-5ca1-43b4-83f7-57bfe805b360","Type":"ContainerStarted","Data":"87aab08108247aaf103e4288a153302ef854d93c64a0f7b44b4955c9566c72d8"} Jul 10 16:36:23.456714 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.456691 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:36:23.578125 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.578054 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs\") pod \"network-metrics-daemon-6c49f\" (UID: \"08a4d929-2c36-40ac-a3e6-8c5caf0c1788\") " pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:23.578250 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:23.578185 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:23.578250 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:23.578244 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs podName:08a4d929-2c36-40ac-a3e6-8c5caf0c1788 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:24.578226626 +0000 UTC m=+4.000103076 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs") pod "network-metrics-daemon-6c49f" (UID: "08a4d929-2c36-40ac-a3e6-8c5caf0c1788") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:23.679337 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.679305 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret\") pod \"global-pull-secret-syncer-lhp9w\" (UID: \"2594870a-c2d7-47e9-8f55-d6c029810a1d\") " pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:23.679486 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:23.679436 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:23.679546 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:23.679508 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret podName:2594870a-c2d7-47e9-8f55-d6c029810a1d nodeName:}" failed. No retries permitted until 2026-07-10 16:36:24.679487649 +0000 UTC m=+4.101364099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret") pod "global-pull-secret-syncer-lhp9w" (UID: "2594870a-c2d7-47e9-8f55-d6c029810a1d") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:23.780226 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.780188 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f99l\" (UniqueName: \"kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l\") pod \"network-check-target-6q44p\" (UID: \"a449c30f-b973-4076-b3da-484943344705\") " pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:23.780380 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:23.780360 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:36:23.780455 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:23.780383 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:36:23.780455 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:23.780396 2585 projected.go:196] Error preparing data for projected volume kube-api-access-8f99l for pod openshift-network-diagnostics/network-check-target-6q44p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:23.780717 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:23.780465 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l podName:a449c30f-b973-4076-b3da-484943344705 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:24.780434729 +0000 UTC m=+4.202311178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8f99l" (UniqueName: "kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l") pod "network-check-target-6q44p" (UID: "a449c30f-b973-4076-b3da-484943344705") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:23.974956 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.974793 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 16:31:23 +0000 UTC" deadline="2028-03-03 22:36:43.657548154 +0000 UTC" Jul 10 16:36:23.974956 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.974854 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14454h0m19.682698244s" Jul 10 16:36:23.983929 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:23.983711 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:36:24.320024 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:24.319936 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:24.320630 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:24.320072 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:24.320630 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:24.320525 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:24.320740 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:24.320633 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:24.587826 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:24.587732 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs\") pod \"network-metrics-daemon-6c49f\" (UID: \"08a4d929-2c36-40ac-a3e6-8c5caf0c1788\") " pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:24.588007 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:24.587917 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:24.588007 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:24.587979 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs podName:08a4d929-2c36-40ac-a3e6-8c5caf0c1788 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:26.587959261 +0000 UTC m=+6.009835713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs") pod "network-metrics-daemon-6c49f" (UID: "08a4d929-2c36-40ac-a3e6-8c5caf0c1788") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:24.688798 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:24.688752 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret\") pod \"global-pull-secret-syncer-lhp9w\" (UID: \"2594870a-c2d7-47e9-8f55-d6c029810a1d\") " pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:24.688985 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:24.688947 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:24.689059 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:24.689008 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret podName:2594870a-c2d7-47e9-8f55-d6c029810a1d nodeName:}" failed. No retries permitted until 2026-07-10 16:36:26.688988158 +0000 UTC m=+6.110864606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret") pod "global-pull-secret-syncer-lhp9w" (UID: "2594870a-c2d7-47e9-8f55-d6c029810a1d") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:24.790263 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:24.790217 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f99l\" (UniqueName: \"kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l\") pod \"network-check-target-6q44p\" (UID: \"a449c30f-b973-4076-b3da-484943344705\") " pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:24.790456 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:24.790403 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:36:24.790456 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:24.790429 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:36:24.790456 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:24.790443 2585 projected.go:196] Error preparing data for projected volume kube-api-access-8f99l for pod openshift-network-diagnostics/network-check-target-6q44p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:24.790603 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:24.790503 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l podName:a449c30f-b973-4076-b3da-484943344705 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:26.790483695 +0000 UTC m=+6.212360161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8f99l" (UniqueName: "kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l") pod "network-check-target-6q44p" (UID: "a449c30f-b973-4076-b3da-484943344705") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:24.975972 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:24.975925 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 16:31:23 +0000 UTC" deadline="2028-02-01 11:58:30.068503866 +0000 UTC" Jul 10 16:36:24.975972 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:24.975970 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13699h22m5.092539236s" Jul 10 16:36:25.322042 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:25.322012 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:25.322898 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:25.322163 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:26.321366 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:26.321104 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:26.321544 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:26.321409 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:26.321625 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:26.321598 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:26.322082 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:26.322037 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:26.382192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:26.382154 2585 generic.go:356] "Generic (PLEG): container finished" podID="22c1e48ab6935889232428e887b9379d" containerID="9b52ede9edcd15c41e92fb886b68ae82a4fca6b5267787004b31242d52e1ea4d" exitCode=0 Jul 10 16:36:26.382374 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:26.382206 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" event={"ID":"22c1e48ab6935889232428e887b9379d","Type":"ContainerDied","Data":"9b52ede9edcd15c41e92fb886b68ae82a4fca6b5267787004b31242d52e1ea4d"} Jul 10 16:36:26.606174 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:26.606002 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs\") pod \"network-metrics-daemon-6c49f\" (UID: \"08a4d929-2c36-40ac-a3e6-8c5caf0c1788\") " pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:26.606344 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:26.606207 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:26.606344 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:26.606279 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs podName:08a4d929-2c36-40ac-a3e6-8c5caf0c1788 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:30.606258534 +0000 UTC m=+10.028134983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs") pod "network-metrics-daemon-6c49f" (UID: "08a4d929-2c36-40ac-a3e6-8c5caf0c1788") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:26.706538 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:26.706500 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret\") pod \"global-pull-secret-syncer-lhp9w\" (UID: \"2594870a-c2d7-47e9-8f55-d6c029810a1d\") " pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:26.706735 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:26.706699 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:26.706842 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:26.706775 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret podName:2594870a-c2d7-47e9-8f55-d6c029810a1d nodeName:}" failed. No retries permitted until 2026-07-10 16:36:30.70675779 +0000 UTC m=+10.128634249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret") pod "global-pull-secret-syncer-lhp9w" (UID: "2594870a-c2d7-47e9-8f55-d6c029810a1d") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:26.807470 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:26.807427 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f99l\" (UniqueName: \"kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l\") pod \"network-check-target-6q44p\" (UID: \"a449c30f-b973-4076-b3da-484943344705\") " pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:26.807645 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:26.807596 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:36:26.807645 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:26.807617 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:36:26.807645 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:26.807630 2585 projected.go:196] Error preparing data for projected volume kube-api-access-8f99l for pod openshift-network-diagnostics/network-check-target-6q44p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:26.807864 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:26.807688 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l podName:a449c30f-b973-4076-b3da-484943344705 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:30.807668728 +0000 UTC m=+10.229545195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8f99l" (UniqueName: "kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l") pod "network-check-target-6q44p" (UID: "a449c30f-b973-4076-b3da-484943344705") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:27.319715 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:27.319682 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:27.320045 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:27.320018 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:28.319933 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:28.319900 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:28.320380 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:28.319955 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:28.320380 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:28.320041 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:28.320380 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:28.320252 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:29.320144 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:29.320050 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:29.320578 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:29.320209 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:30.319939 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:30.319811 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:30.320143 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:30.319959 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:30.320143 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:30.320072 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:30.320578 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:30.320176 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:30.638445 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:30.638347 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs\") pod \"network-metrics-daemon-6c49f\" (UID: \"08a4d929-2c36-40ac-a3e6-8c5caf0c1788\") " pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:30.638610 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:30.638532 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:30.638610 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:30.638596 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs podName:08a4d929-2c36-40ac-a3e6-8c5caf0c1788 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:38.63857785 +0000 UTC m=+18.060454300 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs") pod "network-metrics-daemon-6c49f" (UID: "08a4d929-2c36-40ac-a3e6-8c5caf0c1788") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:30.739615 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:30.739576 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret\") pod \"global-pull-secret-syncer-lhp9w\" (UID: \"2594870a-c2d7-47e9-8f55-d6c029810a1d\") " pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:30.739824 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:30.739766 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:30.739903 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:30.739846 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret podName:2594870a-c2d7-47e9-8f55-d6c029810a1d nodeName:}" failed. No retries permitted until 2026-07-10 16:36:38.739825193 +0000 UTC m=+18.161701643 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret") pod "global-pull-secret-syncer-lhp9w" (UID: "2594870a-c2d7-47e9-8f55-d6c029810a1d") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:30.840573 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:30.840537 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f99l\" (UniqueName: \"kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l\") pod \"network-check-target-6q44p\" (UID: \"a449c30f-b973-4076-b3da-484943344705\") " pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:30.840770 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:30.840734 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:36:30.840860 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:30.840775 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:36:30.840860 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:30.840788 2585 projected.go:196] Error preparing data for projected volume kube-api-access-8f99l for pod openshift-network-diagnostics/network-check-target-6q44p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:30.840860 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:30.840846 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l podName:a449c30f-b973-4076-b3da-484943344705 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:38.840828704 +0000 UTC m=+18.262705153 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8f99l" (UniqueName: "kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l") pod "network-check-target-6q44p" (UID: "a449c30f-b973-4076-b3da-484943344705") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:31.320643 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:31.320252 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:31.320643 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:31.320393 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:32.320192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:32.320115 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:32.320192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:32.320148 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:32.320399 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:32.320260 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:32.320460 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:32.320386 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:33.319790 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:33.319725 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:33.320267 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:33.319898 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:34.319409 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:34.319375 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:34.319603 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:34.319505 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:34.319603 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:34.319558 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:34.319725 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:34.319656 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:35.319580 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:35.319485 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:35.320002 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:35.319636 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:36.319759 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:36.319726 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:36.320180 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:36.319734 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:36.320180 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:36.319856 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:36.320180 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:36.319954 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:37.319344 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:37.319306 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:37.319519 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:37.319458 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:38.319816 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:38.319776 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:38.320331 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:38.319830 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:38.320331 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:38.319910 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:38.320331 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:38.320004 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:38.700434 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:38.700352 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs\") pod \"network-metrics-daemon-6c49f\" (UID: \"08a4d929-2c36-40ac-a3e6-8c5caf0c1788\") " pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:38.700638 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:38.700467 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:38.700638 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:38.700533 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs podName:08a4d929-2c36-40ac-a3e6-8c5caf0c1788 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:54.700512563 +0000 UTC m=+34.122389016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs") pod "network-metrics-daemon-6c49f" (UID: "08a4d929-2c36-40ac-a3e6-8c5caf0c1788") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:38.801001 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:38.800960 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret\") pod \"global-pull-secret-syncer-lhp9w\" (UID: \"2594870a-c2d7-47e9-8f55-d6c029810a1d\") " pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:38.801175 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:38.801130 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:38.801230 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:38.801214 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret podName:2594870a-c2d7-47e9-8f55-d6c029810a1d nodeName:}" failed. No retries permitted until 2026-07-10 16:36:54.801192594 +0000 UTC m=+34.223069056 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret") pod "global-pull-secret-syncer-lhp9w" (UID: "2594870a-c2d7-47e9-8f55-d6c029810a1d") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:38.902162 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:38.902129 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f99l\" (UniqueName: \"kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l\") pod \"network-check-target-6q44p\" (UID: \"a449c30f-b973-4076-b3da-484943344705\") " pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:38.902324 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:38.902299 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:36:38.902369 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:38.902332 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:36:38.902369 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:38.902345 2585 projected.go:196] Error preparing data for projected volume kube-api-access-8f99l for pod openshift-network-diagnostics/network-check-target-6q44p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:38.902435 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:38.902403 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l podName:a449c30f-b973-4076-b3da-484943344705 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:54.902389349 +0000 UTC m=+34.324265794 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8f99l" (UniqueName: "kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l") pod "network-check-target-6q44p" (UID: "a449c30f-b973-4076-b3da-484943344705") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:39.320112 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:39.320077 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:39.320562 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:39.320210 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:40.319735 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:40.319691 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:40.319947 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:40.319737 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:40.319947 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:40.319848 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:40.320044 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:40.319942 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:41.320594 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:41.320560 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:41.321075 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:41.320673 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:42.319650 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:42.319614 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:42.319853 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:42.319620 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:42.319853 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:42.319760 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:42.319853 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:42.319828 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:43.322549 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:43.322521 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:43.322908 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:43.322622 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:44.320475 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.320250 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:44.320617 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.320275 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:44.320617 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:44.320569 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:44.320704 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:44.320618 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:44.416780 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.416725 2585 generic.go:356] "Generic (PLEG): container finished" podID="4a75db21-5ca1-43b4-83f7-57bfe805b360" containerID="6488225af6a2dce1cbfc505b3236b6f3a849e6cb36b44dc448eaa5b1895fa9f4" exitCode=0 Jul 10 16:36:44.417528 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.416782 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fnqd8" event={"ID":"4a75db21-5ca1-43b4-83f7-57bfe805b360","Type":"ContainerDied","Data":"6488225af6a2dce1cbfc505b3236b6f3a849e6cb36b44dc448eaa5b1895fa9f4"} Jul 10 16:36:44.418428 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.418408 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" event={"ID":"fc77b858-26c5-406d-97ab-1dec3ec73678","Type":"ContainerStarted","Data":"e6a43c88808fb8614ee1e2cec3eeb87e6477b8fee0e1f43d3ff04bb48a4b4315"} Jul 10 16:36:44.420302 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.420279 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" event={"ID":"22c1e48ab6935889232428e887b9379d","Type":"ContainerStarted","Data":"26a4e733b0271c696c6a9047992f6826ca72450e92b5b554cc6a5546785c42f9"} Jul 10 16:36:44.421988 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.421973 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 16:36:44.422287 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.422268 2585 generic.go:356] "Generic (PLEG): container finished" podID="b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56" containerID="4dd7d2180a93e531a35ab6b128a60241f786e82863dfafc1d245307983ebf6bb" exitCode=1 Jul 10 16:36:44.422362 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.422324 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" event={"ID":"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56","Type":"ContainerStarted","Data":"f8adc1f920f7c78e513cbab1a4eb34d43a2f3b2d27d234431f1c4c6ffe48b62a"} Jul 10 16:36:44.422411 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.422382 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" event={"ID":"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56","Type":"ContainerDied","Data":"4dd7d2180a93e531a35ab6b128a60241f786e82863dfafc1d245307983ebf6bb"} Jul 10 16:36:44.422450 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.422433 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" event={"ID":"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56","Type":"ContainerStarted","Data":"557dddfe48550e8512f1f971d0fb1cf6294b93ba6d09f55333c71fd4e8977f1c"} Jul 10 16:36:44.423756 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.423723 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal" event={"ID":"e404ee0301f15458abcdaff3158e6ab9","Type":"ContainerStarted","Data":"4b167d893ced8c219eacea979708d129b48185f4a6cce41ac7797223b85e5441"} Jul 10 16:36:44.425160 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.425139 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dsv6h" event={"ID":"1d54c0b4-7530-45fb-804c-777c15ff35eb","Type":"ContainerStarted","Data":"be643971a058711b75bc2320c10784d3cdffdc26006dcb957968351961fceac9"} Jul 10 16:36:44.426374 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.426354 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fmt8m" event={"ID":"405fff59-3355-4cd2-9371-77c04c509283","Type":"ContainerStarted","Data":"cc73f914f494c24fb52c775a260f2baafc323d7b9f2bcc48b8218f29c78bf0f9"} Jul 10 16:36:44.427534 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.427514 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" event={"ID":"cf308df8-1c08-449a-91bf-f24e5485a3b4","Type":"ContainerStarted","Data":"f2cd72d1d0cac71daea39c1ba57ccac4808f69a2c230bdbe0ce4695ba48238aa"} Jul 10 16:36:44.428762 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.428729 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m7vs8" event={"ID":"2d492a8c-19e2-462a-85bc-ac78c1898d73","Type":"ContainerStarted","Data":"cd503b9e505b726fb2d226d28f017bbf2c9399b83cb45d9a904846a330c5e9e9"} Jul 10 16:36:44.430064 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.430044 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nzk6t" event={"ID":"60be53b8-7bc0-456c-9029-76e0f65316e6","Type":"ContainerStarted","Data":"403d5679470c51bf65df1be8b68c44be5f8ca0cda20a5e3a6146c39b22b5cd67"} Jul 10 16:36:44.455051 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.455010 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nzk6t" podStartSLOduration=3.346121799 podStartE2EDuration="23.454993478s" totalImagesPullingTime="20.108871679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:36:44.454780459 +0000 UTC m=+23.876656945" watchObservedRunningTime="2026-07-10 16:36:44.454993478 +0000 UTC m=+23.876869946" Jul 10 16:36:44.491821 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.491778 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-m7vs8" podStartSLOduration=3.384572158 podStartE2EDuration="23.491762918s" totalImagesPullingTime="20.10719076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:36:44.47136702 +0000 UTC m=+23.893243482" watchObservedRunningTime="2026-07-10 16:36:44.491762918 +0000 UTC m=+23.913639376" Jul 10 16:36:44.491930 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.491865 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dsv6h" podStartSLOduration=11.640827484 podStartE2EDuration="23.491855823s" totalImagesPullingTime="11.851028339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:36:44.491323044 +0000 UTC m=+23.913199508" watchObservedRunningTime="2026-07-10 16:36:44.491855823 +0000 UTC m=+23.913732291" Jul 10 16:36:44.505940 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.505842 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" podStartSLOduration=22.50582931 podStartE2EDuration="22.50582931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:36:44.505447063 +0000 UTC m=+23.927323531" watchObservedRunningTime="2026-07-10 16:36:44.50582931 +0000 UTC m=+23.927705774" Jul 10 16:36:44.541437 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.540871 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fmt8m" podStartSLOduration=3.360366225 podStartE2EDuration="23.540853253s" totalImagesPullingTime="20.180487028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:36:44.540382797 +0000 UTC m=+23.962259263" watchObservedRunningTime="2026-07-10 16:36:44.540853253 +0000 UTC m=+23.962729721" Jul 10 16:36:44.541437 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.541090 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal" podStartSLOduration=22.541082719 podStartE2EDuration="22.541082719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:36:44.521626612 +0000 UTC m=+23.943503077" watchObservedRunningTime="2026-07-10 16:36:44.541082719 +0000 UTC m=+23.962959186" Jul 10 16:36:44.559892 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.559850 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s7w2p" podStartSLOduration=3.415401404 podStartE2EDuration="23.559832637s" totalImagesPullingTime="20.144431233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:36:44.559665698 +0000 UTC m=+23.981542163" watchObservedRunningTime="2026-07-10 16:36:44.559832637 +0000 UTC m=+23.981709104" Jul 10 16:36:44.793594 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:44.793569 2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 16:36:45.306268 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:45.306114 2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T16:36:44.793590112Z","UUID":"11fb7cd7-e3f5-45aa-93e3-addbe6f18add","Handler":null,"Name":"","Endpoint":""} Jul 10 16:36:45.307711 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:45.307688 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 16:36:45.307845 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:45.307732 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 16:36:45.320035 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:45.320002 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:45.320204 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:45.320164 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:45.436049 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:45.436019 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 16:36:45.436558 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:45.436530 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" event={"ID":"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56","Type":"ContainerStarted","Data":"7650dc45b7d557357682e7fcd98e5be71a03cb1e4ababc021a6492a36194ac21"} Jul 10 16:36:45.436644 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:45.436624 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" event={"ID":"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56","Type":"ContainerStarted","Data":"25f7caaea36410e0bc1d42f67864284dd1aa6c8444d570c38adb649013fcdeca"} Jul 10 16:36:45.436718 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:45.436683 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" event={"ID":"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56","Type":"ContainerStarted","Data":"8fa2c341703503284eefdd9f6b7639e5af4a469025f3331a999af7e1243f92cb"} Jul 10 16:36:45.438210 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:45.438178 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dn8s4" event={"ID":"696116b2-b4c2-458b-ad03-8b9f224ba24b","Type":"ContainerStarted","Data":"3240dfeeac8e5c6b133535c9a75ef457678d3ed9c00f6d4074e5d2ff8a0fc57b"} Jul 10 16:36:45.440417 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:45.440384 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" event={"ID":"cf308df8-1c08-449a-91bf-f24e5485a3b4","Type":"ContainerStarted","Data":"830591ac4d86ce36b360b01835cd93356135801622e63e60ded1167df1ce1ac6"} Jul 10 16:36:45.453583 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:45.453544 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dn8s4" podStartSLOduration=4.33251101 podStartE2EDuration="24.453530872s" totalImagesPullingTime="20.121019862s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:36:45.453298126 +0000 UTC m=+24.875174592" watchObservedRunningTime="2026-07-10 16:36:45.453530872 +0000 UTC m=+24.875407338" Jul 10 16:36:46.319505 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:46.319475 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:46.319798 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:46.319475 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:46.319798 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:46.319594 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:46.319798 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:46.319694 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:47.319962 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:47.319671 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:47.320526 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:47.320055 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:47.448119 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:47.448075 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 16:36:47.448555 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:47.448530 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" event={"ID":"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56","Type":"ContainerStarted","Data":"979700db5d59ea247b23c9e37dea6599cb3ae86359ad70ec1d2370702cc92911"} Jul 10 16:36:47.450884 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:47.450856 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" event={"ID":"cf308df8-1c08-449a-91bf-f24e5485a3b4","Type":"ContainerStarted","Data":"0f4f198cc9bce7aa0d51aa1c40c68e5e771755995071635663ec3d90ec85a2d7"} Jul 10 16:36:47.469551 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:47.469512 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtzhp" podStartSLOduration=3.559138694 podStartE2EDuration="26.469498924s" totalImagesPullingTime="22.91036023s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:36:47.468970264 +0000 UTC m=+26.890846731" watchObservedRunningTime="2026-07-10 16:36:47.469498924 +0000 UTC m=+26.891375437" Jul 10 16:36:48.319977 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:48.319942 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:48.320437 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:48.319942 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:48.320437 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:48.320065 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:48.320437 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:48.320133 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:48.578580 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:48.578508 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nzk6t" Jul 10 16:36:48.579047 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:48.579029 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nzk6t" Jul 10 16:36:49.323044 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:49.322873 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:49.323690 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:49.323128 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:49.457343 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:49.457228 2585 generic.go:356] "Generic (PLEG): container finished" podID="4a75db21-5ca1-43b4-83f7-57bfe805b360" containerID="505aa3727268865c49f605f0cfe144e0275cf1a06a59a625eeaa9f4a2068aa8a" exitCode=0 Jul 10 16:36:49.457343 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:49.457317 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fnqd8" event={"ID":"4a75db21-5ca1-43b4-83f7-57bfe805b360","Type":"ContainerDied","Data":"505aa3727268865c49f605f0cfe144e0275cf1a06a59a625eeaa9f4a2068aa8a"} Jul 10 16:36:49.460759 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:49.460725 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 16:36:49.461191 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:49.461161 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" event={"ID":"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56","Type":"ContainerStarted","Data":"e6a9ed713b92e67147ebf05fc201578689b3a1879fa843bcf327af8d6f43d317"} Jul 10 16:36:49.461569 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:49.461534 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:49.461664 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:49.461624 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:49.461664 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:49.461651 2585 scope.go:117] "RemoveContainer" containerID="4dd7d2180a93e531a35ab6b128a60241f786e82863dfafc1d245307983ebf6bb" Jul 10 16:36:49.461739 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:49.461672 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:49.478626 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:49.478607 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:49.479127 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:49.479110 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:36:49.798943 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:49.798867 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nzk6t" Jul 10 16:36:49.799425 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:49.799409 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nzk6t" Jul 10 16:36:50.319204 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:50.319173 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:50.319411 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:50.319173 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:50.319411 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:50.319298 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:50.319411 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:50.319358 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:50.467191 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:50.467168 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 16:36:50.467607 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:50.467536 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" event={"ID":"b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56","Type":"ContainerStarted","Data":"0d386224bc44a06622f89212ab7c8aa127ae4a90e12d3550cf08a407f0855637"} Jul 10 16:36:50.496271 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:50.496217 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" podStartSLOduration=8.895722805 podStartE2EDuration="29.496197049s" totalImagesPullingTime="20.600474244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:36:50.494645387 +0000 UTC m=+29.916521883" watchObservedRunningTime="2026-07-10 16:36:50.496197049 +0000 UTC m=+29.918073520" Jul 10 16:36:51.322855 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:51.321946 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:51.322855 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:51.322099 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:51.458958 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:51.458716 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lhp9w"] Jul 10 16:36:51.459120 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:51.459094 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:51.459228 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:51.459205 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:51.459523 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:51.459496 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6c49f"] Jul 10 16:36:51.460379 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:51.460276 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6q44p"] Jul 10 16:36:51.460524 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:51.460383 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:51.460524 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:51.460474 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:51.471946 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:51.471920 2585 generic.go:356] "Generic (PLEG): container finished" podID="4a75db21-5ca1-43b4-83f7-57bfe805b360" containerID="f5c745f8633c4c43a9e394077d697fab0c9177b345e4aaf2f83187c0be0c19a8" exitCode=0 Jul 10 16:36:51.472328 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:51.471997 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:51.472328 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:51.472001 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fnqd8" event={"ID":"4a75db21-5ca1-43b4-83f7-57bfe805b360","Type":"ContainerDied","Data":"f5c745f8633c4c43a9e394077d697fab0c9177b345e4aaf2f83187c0be0c19a8"} Jul 10 16:36:51.472328 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:51.472145 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:53.320202 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:53.320163 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:53.320711 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:53.320215 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:53.320711 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:53.320170 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:53.320711 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:53.320315 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:53.320711 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:53.320342 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:53.320711 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:53.320437 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:53.478219 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:53.478143 2585 generic.go:356] "Generic (PLEG): container finished" podID="4a75db21-5ca1-43b4-83f7-57bfe805b360" containerID="f96e5f14fbf87f269343f1cd3adc94e5150273dbef8c9af72c940e32cf3ea292" exitCode=0 Jul 10 16:36:53.478219 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:53.478190 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fnqd8" event={"ID":"4a75db21-5ca1-43b4-83f7-57bfe805b360","Type":"ContainerDied","Data":"f96e5f14fbf87f269343f1cd3adc94e5150273dbef8c9af72c940e32cf3ea292"} Jul 10 16:36:54.720904 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:54.720870 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs\") pod \"network-metrics-daemon-6c49f\" (UID: \"08a4d929-2c36-40ac-a3e6-8c5caf0c1788\") " pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:54.721608 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:54.720999 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:54.721608 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:54.721070 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs podName:08a4d929-2c36-40ac-a3e6-8c5caf0c1788 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:26.721056046 +0000 UTC m=+66.142932495 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs") pod "network-metrics-daemon-6c49f" (UID: "08a4d929-2c36-40ac-a3e6-8c5caf0c1788") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:54.821699 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:54.821661 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret\") pod \"global-pull-secret-syncer-lhp9w\" (UID: \"2594870a-c2d7-47e9-8f55-d6c029810a1d\") " pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:54.821887 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:54.821832 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:54.821943 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:54.821908 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret podName:2594870a-c2d7-47e9-8f55-d6c029810a1d nodeName:}" failed. No retries permitted until 2026-07-10 16:37:26.821885994 +0000 UTC m=+66.243762440 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret") pod "global-pull-secret-syncer-lhp9w" (UID: "2594870a-c2d7-47e9-8f55-d6c029810a1d") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:54.922771 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:54.922724 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f99l\" (UniqueName: \"kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l\") pod \"network-check-target-6q44p\" (UID: \"a449c30f-b973-4076-b3da-484943344705\") " pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:54.922991 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:54.922899 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:36:54.922991 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:54.922921 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:36:54.922991 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:54.922932 2585 projected.go:196] Error preparing data for projected volume kube-api-access-8f99l for pod openshift-network-diagnostics/network-check-target-6q44p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:54.922991 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:54.922990 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l podName:a449c30f-b973-4076-b3da-484943344705 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:26.922975023 +0000 UTC m=+66.344851468 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8f99l" (UniqueName: "kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l") pod "network-check-target-6q44p" (UID: "a449c30f-b973-4076-b3da-484943344705") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:55.323310 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:55.323281 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:55.323496 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:55.323399 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:55.323496 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:55.323412 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:55.323496 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:55.323428 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:55.323652 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:55.323510 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:55.323652 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:55.323616 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:57.319515 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.319433 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:57.320091 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.319433 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:57.320091 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.319565 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:57.320091 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:57.319585 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:36:57.320091 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:57.319641 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6q44p" podUID="a449c30f-b973-4076-b3da-484943344705" Jul 10 16:36:57.320091 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:57.319759 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhp9w" podUID="2594870a-c2d7-47e9-8f55-d6c029810a1d" Jul 10 16:36:57.388541 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.388512 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeReady" Jul 10 16:36:57.388706 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.388654 2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 16:36:57.435882 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.435851 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kp9dv"] Jul 10 16:36:57.438411 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.438385 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kp9dv" Jul 10 16:36:57.438913 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.438885 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hbr2f"] Jul 10 16:36:57.440811 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.440792 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hbr2f" Jul 10 16:36:57.443760 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.443546 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/32d1ff51-842a-4cb2-b179-ffe84520aad3-config-volume\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:36:57.443859 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.443806 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpc4d\" (UniqueName: \"kubernetes.io/projected/7ea783bf-0092-4036-a31b-bc4111a252c5-kube-api-access-tpc4d\") pod \"ingress-canary-hbr2f\" (UID: \"7ea783bf-0092-4036-a31b-bc4111a252c5\") " pod="openshift-ingress-canary/ingress-canary-hbr2f" Jul 10 16:36:57.443859 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.443851 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32d1ff51-842a-4cb2-b179-ffe84520aad3-tmp-dir\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:36:57.443963 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.443942 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl72j\" (UniqueName: \"kubernetes.io/projected/32d1ff51-842a-4cb2-b179-ffe84520aad3-kube-api-access-xl72j\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:36:57.444016 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.443983 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ea783bf-0092-4036-a31b-bc4111a252c5-cert\") pod \"ingress-canary-hbr2f\" (UID: \"7ea783bf-0092-4036-a31b-bc4111a252c5\") " pod="openshift-ingress-canary/ingress-canary-hbr2f" Jul 10 16:36:57.444016 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.444003 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:36:57.446360 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.446343 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 16:36:57.447175 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.447153 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jqrx8\"" Jul 10 16:36:57.447484 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.447469 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 16:36:57.448133 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.448114 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 16:36:57.451333 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.451314 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 16:36:57.451423 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.451365 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 16:36:57.451679 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.451662 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xrt5s\"" Jul 10 16:36:57.465767 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.465730 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hbr2f"] Jul 10 16:36:57.468243 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.468219 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kp9dv"] Jul 10 16:36:57.544912 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.544879 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32d1ff51-842a-4cb2-b179-ffe84520aad3-tmp-dir\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:36:57.545087 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.544957 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl72j\" (UniqueName: \"kubernetes.io/projected/32d1ff51-842a-4cb2-b179-ffe84520aad3-kube-api-access-xl72j\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:36:57.545087 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.544998 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ea783bf-0092-4036-a31b-bc4111a252c5-cert\") pod \"ingress-canary-hbr2f\" (UID: \"7ea783bf-0092-4036-a31b-bc4111a252c5\") " pod="openshift-ingress-canary/ingress-canary-hbr2f" Jul 10 16:36:57.545087 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.545019 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:36:57.545087 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.545049 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/32d1ff51-842a-4cb2-b179-ffe84520aad3-config-volume\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:36:57.545087 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.545078 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpc4d\" (UniqueName: \"kubernetes.io/projected/7ea783bf-0092-4036-a31b-bc4111a252c5-kube-api-access-tpc4d\") pod \"ingress-canary-hbr2f\" (UID: \"7ea783bf-0092-4036-a31b-bc4111a252c5\") " pod="openshift-ingress-canary/ingress-canary-hbr2f" Jul 10 16:36:57.545374 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.545230 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32d1ff51-842a-4cb2-b179-ffe84520aad3-tmp-dir\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:36:57.545374 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:57.545298 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:36:57.545374 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:57.545355 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls podName:32d1ff51-842a-4cb2-b179-ffe84520aad3 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:58.045335262 +0000 UTC m=+37.467211708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls") pod "dns-default-kp9dv" (UID: "32d1ff51-842a-4cb2-b179-ffe84520aad3") : secret "dns-default-metrics-tls" not found Jul 10 16:36:57.545714 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.545679 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/32d1ff51-842a-4cb2-b179-ffe84520aad3-config-volume\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:36:57.549896 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.549869 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ea783bf-0092-4036-a31b-bc4111a252c5-cert\") pod \"ingress-canary-hbr2f\" (UID: \"7ea783bf-0092-4036-a31b-bc4111a252c5\") " pod="openshift-ingress-canary/ingress-canary-hbr2f" Jul 10 16:36:57.555245 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.555225 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpc4d\" (UniqueName: \"kubernetes.io/projected/7ea783bf-0092-4036-a31b-bc4111a252c5-kube-api-access-tpc4d\") pod \"ingress-canary-hbr2f\" (UID: \"7ea783bf-0092-4036-a31b-bc4111a252c5\") " pod="openshift-ingress-canary/ingress-canary-hbr2f" Jul 10 16:36:57.557190 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.557162 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl72j\" (UniqueName: \"kubernetes.io/projected/32d1ff51-842a-4cb2-b179-ffe84520aad3-kube-api-access-xl72j\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:36:57.757213 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:57.757182 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hbr2f" Jul 10 16:36:58.048328 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:58.048247 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:36:58.048468 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:58.048371 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:36:58.048468 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:58.048427 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls podName:32d1ff51-842a-4cb2-b179-ffe84520aad3 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:59.048411668 +0000 UTC m=+38.470288116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls") pod "dns-default-kp9dv" (UID: "32d1ff51-842a-4cb2-b179-ffe84520aad3") : secret "dns-default-metrics-tls" not found Jul 10 16:36:58.457548 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:58.457528 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hbr2f"] Jul 10 16:36:58.460968 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:36:58.460944 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea783bf_0092_4036_a31b_bc4111a252c5.slice/crio-a125077531be77cf048e0d5c0d51ca844377d715a40a502bf117aed68effcdef WatchSource:0}: Error finding container a125077531be77cf048e0d5c0d51ca844377d715a40a502bf117aed68effcdef: Status 404 returned error can't find the container with id a125077531be77cf048e0d5c0d51ca844377d715a40a502bf117aed68effcdef Jul 10 16:36:58.493691 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:58.493664 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hbr2f" event={"ID":"7ea783bf-0092-4036-a31b-bc4111a252c5","Type":"ContainerStarted","Data":"a125077531be77cf048e0d5c0d51ca844377d715a40a502bf117aed68effcdef"} Jul 10 16:36:58.496669 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:58.496646 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fnqd8" event={"ID":"4a75db21-5ca1-43b4-83f7-57bfe805b360","Type":"ContainerStarted","Data":"649ea0e24c53f500985c5f2c2abee19592eb642d0c0fd0e89f6b8c36105e87f7"} Jul 10 16:36:59.057143 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:59.056945 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:36:59.057328 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:59.057101 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:36:59.057328 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:36:59.057215 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls podName:32d1ff51-842a-4cb2-b179-ffe84520aad3 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:01.057197568 +0000 UTC m=+40.479074013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls") pod "dns-default-kp9dv" (UID: "32d1ff51-842a-4cb2-b179-ffe84520aad3") : secret "dns-default-metrics-tls" not found Jul 10 16:36:59.319882 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:59.319806 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:36:59.320043 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:59.319806 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:36:59.320499 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:59.320204 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:36:59.322985 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:59.322868 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 16:36:59.324174 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:59.324149 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ggg2b\"" Jul 10 16:36:59.324174 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:59.324165 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 16:36:59.324324 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:59.324155 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 16:36:59.324324 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:59.324293 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bc5mm\"" Jul 10 16:36:59.324422 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:59.324381 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 16:36:59.502600 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:59.502568 2585 generic.go:356] "Generic (PLEG): container finished" podID="4a75db21-5ca1-43b4-83f7-57bfe805b360" containerID="649ea0e24c53f500985c5f2c2abee19592eb642d0c0fd0e89f6b8c36105e87f7" exitCode=0 Jul 10 16:36:59.503028 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:36:59.502620 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fnqd8" event={"ID":"4a75db21-5ca1-43b4-83f7-57bfe805b360","Type":"ContainerDied","Data":"649ea0e24c53f500985c5f2c2abee19592eb642d0c0fd0e89f6b8c36105e87f7"} Jul 10 16:37:00.508985 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:00.508948 2585 generic.go:356] "Generic (PLEG): container finished" podID="4a75db21-5ca1-43b4-83f7-57bfe805b360" containerID="14b02023ecd02d466e04a45f74d59cc2624b4cd789e8d684037cad6a5afcc387" exitCode=0 Jul 10 16:37:00.509514 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:00.509003 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fnqd8" event={"ID":"4a75db21-5ca1-43b4-83f7-57bfe805b360","Type":"ContainerDied","Data":"14b02023ecd02d466e04a45f74d59cc2624b4cd789e8d684037cad6a5afcc387"} Jul 10 16:37:01.071109 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:01.071032 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:37:01.071231 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:37:01.071140 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:37:01.071231 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:37:01.071190 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls podName:32d1ff51-842a-4cb2-b179-ffe84520aad3 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:05.071177005 +0000 UTC m=+44.493053449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls") pod "dns-default-kp9dv" (UID: "32d1ff51-842a-4cb2-b179-ffe84520aad3") : secret "dns-default-metrics-tls" not found Jul 10 16:37:01.514525 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:01.514493 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fnqd8" event={"ID":"4a75db21-5ca1-43b4-83f7-57bfe805b360","Type":"ContainerStarted","Data":"747f30a3bcb14f03532d1a6cf7da8239026d716301f4b5fd11f70b6f0a855eba"} Jul 10 16:37:01.515862 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:01.515839 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hbr2f" event={"ID":"7ea783bf-0092-4036-a31b-bc4111a252c5","Type":"ContainerStarted","Data":"d55181fcdf632b0d3c663f2805b5320ed07726c71e3e8d039e7de2764bfcc831"} Jul 10 16:37:01.538939 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:01.538792 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fnqd8" podStartSLOduration=8.533074734 podStartE2EDuration="40.538775418s" totalImagesPullingTime="32.005700684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:37:01.537181857 +0000 UTC m=+40.959058328" watchObservedRunningTime="2026-07-10 16:37:01.538775418 +0000 UTC m=+40.960651885" Jul 10 16:37:01.552996 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:01.552961 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hbr2f" podStartSLOduration=2.249772395 podStartE2EDuration="4.552951876s" totalImagesPullingTime="2.303179481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:36:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:37:01.552500071 +0000 UTC m=+40.974376539" watchObservedRunningTime="2026-07-10 16:37:01.552951876 +0000 UTC m=+40.974828343" Jul 10 16:37:05.099054 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:05.099008 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:37:05.099429 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:37:05.099126 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:37:05.099429 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:37:05.099177 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls podName:32d1ff51-842a-4cb2-b179-ffe84520aad3 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:13.099162956 +0000 UTC m=+52.521039401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls") pod "dns-default-kp9dv" (UID: "32d1ff51-842a-4cb2-b179-ffe84520aad3") : secret "dns-default-metrics-tls" not found Jul 10 16:37:13.157638 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:13.157591 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:37:13.158133 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:37:13.157766 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:37:13.158133 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:37:13.157837 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls podName:32d1ff51-842a-4cb2-b179-ffe84520aad3 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:29.157818429 +0000 UTC m=+68.579694873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls") pod "dns-default-kp9dv" (UID: "32d1ff51-842a-4cb2-b179-ffe84520aad3") : secret "dns-default-metrics-tls" not found Jul 10 16:37:21.483208 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:21.483166 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4lqv6" Jul 10 16:37:26.746947 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:26.746915 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs\") pod \"network-metrics-daemon-6c49f\" (UID: \"08a4d929-2c36-40ac-a3e6-8c5caf0c1788\") " pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:37:26.749871 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:26.749856 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 16:37:26.757101 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:37:26.757082 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 16:37:26.757160 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:37:26.757150 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs podName:08a4d929-2c36-40ac-a3e6-8c5caf0c1788 nodeName:}" failed. No retries permitted until 2026-07-10 16:38:30.7571353 +0000 UTC m=+130.179011744 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs") pod "network-metrics-daemon-6c49f" (UID: "08a4d929-2c36-40ac-a3e6-8c5caf0c1788") : secret "metrics-daemon-secret" not found Jul 10 16:37:26.847671 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:26.847644 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret\") pod \"global-pull-secret-syncer-lhp9w\" (UID: \"2594870a-c2d7-47e9-8f55-d6c029810a1d\") " pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:37:26.851954 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:26.851939 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 16:37:26.861287 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:26.861265 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2594870a-c2d7-47e9-8f55-d6c029810a1d-original-pull-secret\") pod \"global-pull-secret-syncer-lhp9w\" (UID: \"2594870a-c2d7-47e9-8f55-d6c029810a1d\") " pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:37:26.932676 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:26.932657 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhp9w" Jul 10 16:37:26.948861 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:26.948838 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f99l\" (UniqueName: \"kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l\") pod \"network-check-target-6q44p\" (UID: \"a449c30f-b973-4076-b3da-484943344705\") " pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:37:26.955153 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:26.952461 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 16:37:26.961880 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:26.961862 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 16:37:26.971681 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:26.971657 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f99l\" (UniqueName: \"kubernetes.io/projected/a449c30f-b973-4076-b3da-484943344705-kube-api-access-8f99l\") pod \"network-check-target-6q44p\" (UID: \"a449c30f-b973-4076-b3da-484943344705\") " pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:37:27.047897 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:27.047867 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lhp9w"] Jul 10 16:37:27.051385 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:37:27.051359 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2594870a_c2d7_47e9_8f55_d6c029810a1d.slice/crio-cb5cb844acb7742dc59777dd6f0a95698b19c161a89a0b3297754fb4a0ef9d47 WatchSource:0}: Error finding container cb5cb844acb7742dc59777dd6f0a95698b19c161a89a0b3297754fb4a0ef9d47: Status 404 returned error can't find the container with id cb5cb844acb7742dc59777dd6f0a95698b19c161a89a0b3297754fb4a0ef9d47 Jul 10 16:37:27.242590 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:27.242564 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ggg2b\"" Jul 10 16:37:27.249951 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:27.249931 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:37:27.365729 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:27.365698 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6q44p"] Jul 10 16:37:27.369255 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:37:27.369231 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda449c30f_b973_4076_b3da_484943344705.slice/crio-4b0d38c9e204a12f27f6a31086805c61a419f9aa84ee5bab1034308fc09a238e WatchSource:0}: Error finding container 4b0d38c9e204a12f27f6a31086805c61a419f9aa84ee5bab1034308fc09a238e: Status 404 returned error can't find the container with id 4b0d38c9e204a12f27f6a31086805c61a419f9aa84ee5bab1034308fc09a238e Jul 10 16:37:27.574828 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:27.574728 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6q44p" event={"ID":"a449c30f-b973-4076-b3da-484943344705","Type":"ContainerStarted","Data":"4b0d38c9e204a12f27f6a31086805c61a419f9aa84ee5bab1034308fc09a238e"} Jul 10 16:37:27.575991 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:27.575962 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lhp9w" event={"ID":"2594870a-c2d7-47e9-8f55-d6c029810a1d","Type":"ContainerStarted","Data":"cb5cb844acb7742dc59777dd6f0a95698b19c161a89a0b3297754fb4a0ef9d47"} Jul 10 16:37:29.166027 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:29.165982 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:37:29.166508 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:37:29.166166 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:37:29.166508 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:37:29.166249 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls podName:32d1ff51-842a-4cb2-b179-ffe84520aad3 nodeName:}" failed. No retries permitted until 2026-07-10 16:38:01.166227045 +0000 UTC m=+100.588103491 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls") pod "dns-default-kp9dv" (UID: "32d1ff51-842a-4cb2-b179-ffe84520aad3") : secret "dns-default-metrics-tls" not found Jul 10 16:37:32.591411 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:32.591314 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6q44p" event={"ID":"a449c30f-b973-4076-b3da-484943344705","Type":"ContainerStarted","Data":"b87f3b52fc642fb1d7ab2f410f025ceaecc1452c9cd001aef7ee6609e76d9617"} Jul 10 16:37:32.591411 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:32.591406 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:37:32.592800 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:32.592775 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lhp9w" event={"ID":"2594870a-c2d7-47e9-8f55-d6c029810a1d","Type":"ContainerStarted","Data":"7ae908dac4c368dccc2ae31bc6ffc51d308516ea4cb85496f3a7eff9e1a72f11"} Jul 10 16:37:32.610669 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:32.610629 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6q44p" podStartSLOduration=66.774276607 podStartE2EDuration="1m11.610616941s" totalImagesPullingTime="4.836340334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:37:32.609122324 +0000 UTC m=+72.030998845" watchObservedRunningTime="2026-07-10 16:37:32.610616941 +0000 UTC m=+72.032493406" Jul 10 16:37:32.624208 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:37:32.624169 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lhp9w" podStartSLOduration=66.456436312 podStartE2EDuration="1m11.624156475s" totalImagesPullingTime="5.167720163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:37:32.623802021 +0000 UTC m=+72.045678507" watchObservedRunningTime="2026-07-10 16:37:32.624156475 +0000 UTC m=+72.046032940" Jul 10 16:38:01.167061 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:01.167023 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:38:01.167477 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:38:01.167144 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:38:01.167477 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:38:01.167204 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls podName:32d1ff51-842a-4cb2-b179-ffe84520aad3 nodeName:}" failed. No retries permitted until 2026-07-10 16:39:05.167187296 +0000 UTC m=+164.589063741 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls") pod "dns-default-kp9dv" (UID: "32d1ff51-842a-4cb2-b179-ffe84520aad3") : secret "dns-default-metrics-tls" not found Jul 10 16:38:03.597726 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:03.597700 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6q44p" Jul 10 16:38:03.962392 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:03.962361 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9"] Jul 10 16:38:03.969598 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:03.969575 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9" Jul 10 16:38:03.972431 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:03.972405 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 16:38:03.972535 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:03.972414 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-sqskc\"" Jul 10 16:38:03.973639 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:03.973623 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 10 16:38:03.973759 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:03.973670 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 16:38:03.973759 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:03.973668 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 16:38:03.978611 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:03.978507 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9"] Jul 10 16:38:04.089078 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:04.089038 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f276efe3-cc31-4f44-be95-e712754966ec-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9\" (UID: \"f276efe3-cc31-4f44-be95-e712754966ec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9" Jul 10 16:38:04.089246 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:04.089095 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd9hm\" (UniqueName: \"kubernetes.io/projected/f276efe3-cc31-4f44-be95-e712754966ec-kube-api-access-vd9hm\") pod \"managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9\" (UID: \"f276efe3-cc31-4f44-be95-e712754966ec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9" Jul 10 16:38:04.190399 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:04.190360 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f276efe3-cc31-4f44-be95-e712754966ec-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9\" (UID: \"f276efe3-cc31-4f44-be95-e712754966ec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9" Jul 10 16:38:04.190555 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:04.190424 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd9hm\" (UniqueName: \"kubernetes.io/projected/f276efe3-cc31-4f44-be95-e712754966ec-kube-api-access-vd9hm\") pod \"managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9\" (UID: \"f276efe3-cc31-4f44-be95-e712754966ec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9" Jul 10 16:38:04.192896 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:04.192869 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f276efe3-cc31-4f44-be95-e712754966ec-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9\" (UID: \"f276efe3-cc31-4f44-be95-e712754966ec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9" Jul 10 16:38:04.199109 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:04.199088 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd9hm\" (UniqueName: \"kubernetes.io/projected/f276efe3-cc31-4f44-be95-e712754966ec-kube-api-access-vd9hm\") pod \"managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9\" (UID: \"f276efe3-cc31-4f44-be95-e712754966ec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9" Jul 10 16:38:04.291731 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:04.291643 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9" Jul 10 16:38:04.430850 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:04.430817 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9"] Jul 10 16:38:04.434114 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:38:04.434091 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf276efe3_cc31_4f44_be95_e712754966ec.slice/crio-d2e32a0742b08443317d9b7147e7e6757cfda85ada9c28a436256bfdad5f2c15 WatchSource:0}: Error finding container d2e32a0742b08443317d9b7147e7e6757cfda85ada9c28a436256bfdad5f2c15: Status 404 returned error can't find the container with id d2e32a0742b08443317d9b7147e7e6757cfda85ada9c28a436256bfdad5f2c15 Jul 10 16:38:04.667165 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:04.667136 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9" event={"ID":"f276efe3-cc31-4f44-be95-e712754966ec","Type":"ContainerStarted","Data":"d2e32a0742b08443317d9b7147e7e6757cfda85ada9c28a436256bfdad5f2c15"} Jul 10 16:38:07.677603 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:07.677573 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9" event={"ID":"f276efe3-cc31-4f44-be95-e712754966ec","Type":"ContainerStarted","Data":"2bfa74d1c1bf74cb1736219045033df16e8ac1da4f5d25cf188e4d3bef8279ec"} Jul 10 16:38:07.693920 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:07.693862 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9" podStartSLOduration=1.650703487 podStartE2EDuration="4.693847433s" totalImagesPullingTime="3.043143946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:38:07.692319684 +0000 UTC m=+107.114196151" watchObservedRunningTime="2026-07-10 16:38:07.693847433 +0000 UTC m=+107.115723900" Jul 10 16:38:30.760541 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:38:30.760511 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs\") pod \"network-metrics-daemon-6c49f\" (UID: \"08a4d929-2c36-40ac-a3e6-8c5caf0c1788\") " pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:38:30.760952 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:38:30.760642 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 16:38:30.760952 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:38:30.760701 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs podName:08a4d929-2c36-40ac-a3e6-8c5caf0c1788 nodeName:}" failed. No retries permitted until 2026-07-10 16:40:32.760686573 +0000 UTC m=+252.182563019 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs") pod "network-metrics-daemon-6c49f" (UID: "08a4d929-2c36-40ac-a3e6-8c5caf0c1788") : secret "metrics-daemon-secret" not found Jul 10 16:39:00.451354 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:39:00.451316 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-kp9dv" podUID="32d1ff51-842a-4cb2-b179-ffe84520aad3" Jul 10 16:39:00.801134 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:39:00.801110 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kp9dv" Jul 10 16:39:02.344957 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:39:02.344922 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:39:05.176072 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:39:05.176032 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:39:05.176440 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:39:05.176184 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:39:05.176440 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:39:05.176268 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls podName:32d1ff51-842a-4cb2-b179-ffe84520aad3 nodeName:}" failed. No retries permitted until 2026-07-10 16:41:07.176250269 +0000 UTC m=+286.598126714 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls") pod "dns-default-kp9dv" (UID: "32d1ff51-842a-4cb2-b179-ffe84520aad3") : secret "dns-default-metrics-tls" not found Jul 10 16:39:16.320119 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:39:16.320036 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:40:07.957558 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:07.957524 2585 generic.go:356] "Generic (PLEG): container finished" podID="f276efe3-cc31-4f44-be95-e712754966ec" containerID="2bfa74d1c1bf74cb1736219045033df16e8ac1da4f5d25cf188e4d3bef8279ec" exitCode=255 Jul 10 16:40:07.957972 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:07.957590 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9" event={"ID":"f276efe3-cc31-4f44-be95-e712754966ec","Type":"ContainerDied","Data":"2bfa74d1c1bf74cb1736219045033df16e8ac1da4f5d25cf188e4d3bef8279ec"} Jul 10 16:40:07.963034 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:07.963018 2585 scope.go:117] "RemoveContainer" containerID="2bfa74d1c1bf74cb1736219045033df16e8ac1da4f5d25cf188e4d3bef8279ec" Jul 10 16:40:08.962511 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:08.962472 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b74ff8d8d-8hvt9" event={"ID":"f276efe3-cc31-4f44-be95-e712754966ec","Type":"ContainerStarted","Data":"b13cb7106372e153fb228e3758f18474c0d6e8d92064cee9176b1a3566f4ddf4"} Jul 10 16:40:32.248389 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.248351 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d74bffcbd-cvthk"] Jul 10 16:40:32.251295 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.251274 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7b9b6c86cb-knlg6"] Jul 10 16:40:32.251428 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.251389 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.253921 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.253902 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:32.254547 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.254529 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 16:40:32.254651 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.254630 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 16:40:32.254715 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.254696 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kj2p2\"" Jul 10 16:40:32.255155 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.255141 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 16:40:32.260435 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.260417 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 16:40:32.263889 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.263865 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 10 16:40:32.264012 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.263889 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7x729\"" Jul 10 16:40:32.264012 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.263930 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 10 16:40:32.264218 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.264202 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 16:40:32.265009 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.264992 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 16:40:32.266797 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.266780 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 10 16:40:32.267401 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.267386 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 10 16:40:32.273999 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.273977 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7b9b6c86cb-knlg6"] Jul 10 16:40:32.278565 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.278548 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d74bffcbd-cvthk"] Jul 10 16:40:32.319681 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.319649 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2411ecc8-9b66-4410-a942-74fd98224442-image-registry-private-configuration\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.319835 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.319693 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2411ecc8-9b66-4410-a942-74fd98224442-ca-trust-extracted\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.319835 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.319740 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-metrics-certs\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:32.319835 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.319773 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-bound-sa-token\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.319958 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.319831 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2411ecc8-9b66-4410-a942-74fd98224442-registry-certificates\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.319958 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.319860 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2411ecc8-9b66-4410-a942-74fd98224442-trusted-ca\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.319958 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.319878 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-default-certificate\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:32.319958 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.319897 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-stats-auth\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:32.319958 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.319921 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkmm4\" (UniqueName: \"kubernetes.io/projected/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-kube-api-access-tkmm4\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:32.319958 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.319954 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2411ecc8-9b66-4410-a942-74fd98224442-installation-pull-secrets\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.320164 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.319978 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.320164 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.320001 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqzsr\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-kube-api-access-sqzsr\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.320164 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.320030 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:32.345510 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.345480 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf"] Jul 10 16:40:32.348462 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.348446 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf" Jul 10 16:40:32.352125 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.352100 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 10 16:40:32.352227 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.352155 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 10 16:40:32.352278 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.352235 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 10 16:40:32.352843 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.352826 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 10 16:40:32.352929 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.352827 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-pd5t4\"" Jul 10 16:40:32.363423 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.363405 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf"] Jul 10 16:40:32.421201 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.421175 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2411ecc8-9b66-4410-a942-74fd98224442-image-registry-private-configuration\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.421286 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.421214 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2411ecc8-9b66-4410-a942-74fd98224442-ca-trust-extracted\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.421286 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.421234 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-metrics-certs\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:32.421374 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:32.421310 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 16:40:32.421374 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.421341 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6f755a9-cc53-4a26-a3ea-8054a806bf74-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-jf7sf\" (UID: \"f6f755a9-cc53-4a26-a3ea-8054a806bf74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf" Jul 10 16:40:32.421448 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.421379 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-bound-sa-token\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.421448 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:32.421406 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-metrics-certs podName:6a5712a8-cdf6-4acc-afb0-21827ce7bbf4 nodeName:}" failed. No retries permitted until 2026-07-10 16:40:32.921388017 +0000 UTC m=+252.343264479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-metrics-certs") pod "router-default-7b9b6c86cb-knlg6" (UID: "6a5712a8-cdf6-4acc-afb0-21827ce7bbf4") : secret "router-metrics-certs-default" not found Jul 10 16:40:32.421448 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.421433 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6f755a9-cc53-4a26-a3ea-8054a806bf74-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-jf7sf\" (UID: \"f6f755a9-cc53-4a26-a3ea-8054a806bf74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf" Jul 10 16:40:32.421574 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.421459 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kccf\" (UniqueName: \"kubernetes.io/projected/f6f755a9-cc53-4a26-a3ea-8054a806bf74-kube-api-access-5kccf\") pod \"kube-storage-version-migrator-operator-799cb7f958-jf7sf\" (UID: \"f6f755a9-cc53-4a26-a3ea-8054a806bf74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf" Jul 10 16:40:32.421574 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.421497 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2411ecc8-9b66-4410-a942-74fd98224442-registry-certificates\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.421574 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.421535 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2411ecc8-9b66-4410-a942-74fd98224442-trusted-ca\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.421574 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.421569 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-default-certificate\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:32.421835 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.421593 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2411ecc8-9b66-4410-a942-74fd98224442-ca-trust-extracted\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.421835 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.421594 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-stats-auth\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:32.421835 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.421654 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkmm4\" (UniqueName: \"kubernetes.io/projected/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-kube-api-access-tkmm4\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:32.421835 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.421680 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2411ecc8-9b66-4410-a942-74fd98224442-installation-pull-secrets\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.421835 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.421699 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.421835 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.421723 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqzsr\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-kube-api-access-sqzsr\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.421835 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.421791 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:32.422175 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:32.421930 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle podName:6a5712a8-cdf6-4acc-afb0-21827ce7bbf4 nodeName:}" failed. No retries permitted until 2026-07-10 16:40:32.921913213 +0000 UTC m=+252.343789664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle") pod "router-default-7b9b6c86cb-knlg6" (UID: "6a5712a8-cdf6-4acc-afb0-21827ce7bbf4") : configmap references non-existent config key: service-ca.crt Jul 10 16:40:32.422175 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.422057 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2411ecc8-9b66-4410-a942-74fd98224442-registry-certificates\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.422306 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:32.422212 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 16:40:32.422306 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:32.422227 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d74bffcbd-cvthk: secret "image-registry-tls" not found Jul 10 16:40:32.422306 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:32.422271 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls podName:2411ecc8-9b66-4410-a942-74fd98224442 nodeName:}" failed. No retries permitted until 2026-07-10 16:40:32.922256768 +0000 UTC m=+252.344133214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls") pod "image-registry-7d74bffcbd-cvthk" (UID: "2411ecc8-9b66-4410-a942-74fd98224442") : secret "image-registry-tls" not found Jul 10 16:40:32.422964 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.422943 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2411ecc8-9b66-4410-a942-74fd98224442-trusted-ca\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.423901 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.423883 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2411ecc8-9b66-4410-a942-74fd98224442-image-registry-private-configuration\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.424131 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.424112 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2411ecc8-9b66-4410-a942-74fd98224442-installation-pull-secrets\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.424375 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.424357 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-stats-auth\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:32.424439 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.424372 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-default-certificate\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:32.432371 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.432346 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-bound-sa-token\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.432853 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.432833 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqzsr\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-kube-api-access-sqzsr\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.432918 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.432901 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkmm4\" (UniqueName: \"kubernetes.io/projected/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-kube-api-access-tkmm4\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:32.522223 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.522169 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6f755a9-cc53-4a26-a3ea-8054a806bf74-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-jf7sf\" (UID: \"f6f755a9-cc53-4a26-a3ea-8054a806bf74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf" Jul 10 16:40:32.522223 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.522197 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6f755a9-cc53-4a26-a3ea-8054a806bf74-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-jf7sf\" (UID: \"f6f755a9-cc53-4a26-a3ea-8054a806bf74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf" Jul 10 16:40:32.522223 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.522214 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kccf\" (UniqueName: \"kubernetes.io/projected/f6f755a9-cc53-4a26-a3ea-8054a806bf74-kube-api-access-5kccf\") pod \"kube-storage-version-migrator-operator-799cb7f958-jf7sf\" (UID: \"f6f755a9-cc53-4a26-a3ea-8054a806bf74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf" Jul 10 16:40:32.522623 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.522606 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6f755a9-cc53-4a26-a3ea-8054a806bf74-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-jf7sf\" (UID: \"f6f755a9-cc53-4a26-a3ea-8054a806bf74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf" Jul 10 16:40:32.523982 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.523966 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6f755a9-cc53-4a26-a3ea-8054a806bf74-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-jf7sf\" (UID: \"f6f755a9-cc53-4a26-a3ea-8054a806bf74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf" Jul 10 16:40:32.529785 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.529760 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kccf\" (UniqueName: \"kubernetes.io/projected/f6f755a9-cc53-4a26-a3ea-8054a806bf74-kube-api-access-5kccf\") pod \"kube-storage-version-migrator-operator-799cb7f958-jf7sf\" (UID: \"f6f755a9-cc53-4a26-a3ea-8054a806bf74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf" Jul 10 16:40:32.656805 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.656782 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf" Jul 10 16:40:32.773546 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.773463 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf"] Jul 10 16:40:32.777328 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:40:32.777302 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f755a9_cc53_4a26_a3ea_8054a806bf74.slice/crio-911c142ec201ccf0a8b97c4733eb0d3a7d39808e7e6e519d8114d59d9379c245 WatchSource:0}: Error finding container 911c142ec201ccf0a8b97c4733eb0d3a7d39808e7e6e519d8114d59d9379c245: Status 404 returned error can't find the container with id 911c142ec201ccf0a8b97c4733eb0d3a7d39808e7e6e519d8114d59d9379c245 Jul 10 16:40:32.823546 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.823522 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs\") pod \"network-metrics-daemon-6c49f\" (UID: \"08a4d929-2c36-40ac-a3e6-8c5caf0c1788\") " pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:40:32.823646 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:32.823634 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 16:40:32.823695 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:32.823689 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs podName:08a4d929-2c36-40ac-a3e6-8c5caf0c1788 nodeName:}" failed. No retries permitted until 2026-07-10 16:42:34.823676301 +0000 UTC m=+374.245552745 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs") pod "network-metrics-daemon-6c49f" (UID: "08a4d929-2c36-40ac-a3e6-8c5caf0c1788") : secret "metrics-daemon-secret" not found Jul 10 16:40:32.924387 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.924365 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-metrics-certs\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:32.924459 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.924404 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:32.924459 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:32.924433 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:32.924581 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:32.924495 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 16:40:32.924581 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:32.924528 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 16:40:32.924581 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:32.924548 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d74bffcbd-cvthk: secret "image-registry-tls" not found Jul 10 16:40:32.924581 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:32.924557 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle podName:6a5712a8-cdf6-4acc-afb0-21827ce7bbf4 nodeName:}" failed. No retries permitted until 2026-07-10 16:40:33.924541459 +0000 UTC m=+253.346417917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle") pod "router-default-7b9b6c86cb-knlg6" (UID: "6a5712a8-cdf6-4acc-afb0-21827ce7bbf4") : configmap references non-existent config key: service-ca.crt Jul 10 16:40:32.924581 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:32.924575 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-metrics-certs podName:6a5712a8-cdf6-4acc-afb0-21827ce7bbf4 nodeName:}" failed. No retries permitted until 2026-07-10 16:40:33.924569158 +0000 UTC m=+253.346445603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-metrics-certs") pod "router-default-7b9b6c86cb-knlg6" (UID: "6a5712a8-cdf6-4acc-afb0-21827ce7bbf4") : secret "router-metrics-certs-default" not found Jul 10 16:40:32.924770 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:32.924598 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls podName:2411ecc8-9b66-4410-a942-74fd98224442 nodeName:}" failed. No retries permitted until 2026-07-10 16:40:33.924579907 +0000 UTC m=+253.346456355 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls") pod "image-registry-7d74bffcbd-cvthk" (UID: "2411ecc8-9b66-4410-a942-74fd98224442") : secret "image-registry-tls" not found Jul 10 16:40:33.029251 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:33.029188 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf" event={"ID":"f6f755a9-cc53-4a26-a3ea-8054a806bf74","Type":"ContainerStarted","Data":"911c142ec201ccf0a8b97c4733eb0d3a7d39808e7e6e519d8114d59d9379c245"} Jul 10 16:40:33.932552 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:33.932511 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:33.933030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:33.932607 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-metrics-certs\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:33.933030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:33.932663 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:33.933030 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:33.932694 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle podName:6a5712a8-cdf6-4acc-afb0-21827ce7bbf4 nodeName:}" failed. No retries permitted until 2026-07-10 16:40:35.932671771 +0000 UTC m=+255.354548219 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle") pod "router-default-7b9b6c86cb-knlg6" (UID: "6a5712a8-cdf6-4acc-afb0-21827ce7bbf4") : configmap references non-existent config key: service-ca.crt Jul 10 16:40:33.933030 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:33.932759 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 16:40:33.933030 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:33.932776 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d74bffcbd-cvthk: secret "image-registry-tls" not found Jul 10 16:40:33.933030 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:33.932818 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls podName:2411ecc8-9b66-4410-a942-74fd98224442 nodeName:}" failed. No retries permitted until 2026-07-10 16:40:35.932803598 +0000 UTC m=+255.354680061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls") pod "image-registry-7d74bffcbd-cvthk" (UID: "2411ecc8-9b66-4410-a942-74fd98224442") : secret "image-registry-tls" not found Jul 10 16:40:33.933030 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:33.932760 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 16:40:33.933030 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:33.932867 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-metrics-certs podName:6a5712a8-cdf6-4acc-afb0-21827ce7bbf4 nodeName:}" failed. No retries permitted until 2026-07-10 16:40:35.932857508 +0000 UTC m=+255.354733958 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-metrics-certs") pod "router-default-7b9b6c86cb-knlg6" (UID: "6a5712a8-cdf6-4acc-afb0-21827ce7bbf4") : secret "router-metrics-certs-default" not found Jul 10 16:40:35.946434 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:35.946407 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-metrics-certs\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:35.946833 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:35.946449 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:35.946833 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:35.946478 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:35.946833 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:35.946553 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 16:40:35.946833 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:35.946577 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle podName:6a5712a8-cdf6-4acc-afb0-21827ce7bbf4 nodeName:}" failed. No retries permitted until 2026-07-10 16:40:39.946563021 +0000 UTC m=+259.368439467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle") pod "router-default-7b9b6c86cb-knlg6" (UID: "6a5712a8-cdf6-4acc-afb0-21827ce7bbf4") : configmap references non-existent config key: service-ca.crt Jul 10 16:40:35.946833 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:35.946581 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 16:40:35.946833 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:35.946600 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d74bffcbd-cvthk: secret "image-registry-tls" not found Jul 10 16:40:35.946833 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:35.946610 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-metrics-certs podName:6a5712a8-cdf6-4acc-afb0-21827ce7bbf4 nodeName:}" failed. No retries permitted until 2026-07-10 16:40:39.946598096 +0000 UTC m=+259.368474541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-metrics-certs") pod "router-default-7b9b6c86cb-knlg6" (UID: "6a5712a8-cdf6-4acc-afb0-21827ce7bbf4") : secret "router-metrics-certs-default" not found Jul 10 16:40:35.946833 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:35.946642 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls podName:2411ecc8-9b66-4410-a942-74fd98224442 nodeName:}" failed. No retries permitted until 2026-07-10 16:40:39.946627846 +0000 UTC m=+259.368504296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls") pod "image-registry-7d74bffcbd-cvthk" (UID: "2411ecc8-9b66-4410-a942-74fd98224442") : secret "image-registry-tls" not found Jul 10 16:40:36.040422 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:36.040397 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf" event={"ID":"f6f755a9-cc53-4a26-a3ea-8054a806bf74","Type":"ContainerStarted","Data":"d47c769d9eca1bd01a4ebd82d81f779d2541691184bf8bbcba8d2331cb472463"} Jul 10 16:40:36.057152 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:36.057104 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf" podStartSLOduration=1.436800013 podStartE2EDuration="4.057091589s" totalImagesPullingTime="2.620291576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:40:36.056964631 +0000 UTC m=+255.478841100" watchObservedRunningTime="2026-07-10 16:40:36.057091589 +0000 UTC m=+255.478968055" Jul 10 16:40:37.508025 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.507986 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4bncj"] Jul 10 16:40:37.510967 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.510950 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:37.514079 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.514055 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 16:40:37.514224 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.514055 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 16:40:37.514404 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.514369 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 16:40:37.514507 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.514491 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 16:40:37.516001 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.515975 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2jlt9\"" Jul 10 16:40:37.524401 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.524384 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4bncj"] Jul 10 16:40:37.557526 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.557501 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-crio-socket\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:37.557617 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.557536 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:37.557617 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.557607 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:37.557687 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.557640 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-data-volume\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:37.557687 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.557656 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9ttj\" (UniqueName: \"kubernetes.io/projected/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-api-access-h9ttj\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:37.658935 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.658900 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-crio-socket\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:37.658935 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.658946 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:37.659156 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.658994 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:37.659156 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.659015 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-data-volume\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:37.659156 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.659030 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9ttj\" (UniqueName: \"kubernetes.io/projected/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-api-access-h9ttj\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:37.659156 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.659034 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-crio-socket\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:37.659156 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:37.659106 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 16:40:37.659156 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:37.659111 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 16:40:37.659156 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:37.659158 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-rbac-proxy-cm podName:8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d nodeName:}" failed. No retries permitted until 2026-07-10 16:40:38.159140304 +0000 UTC m=+257.581016750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4bncj" (UID: "8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d") : configmap "kube-rbac-proxy" not found Jul 10 16:40:37.659454 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:37.659190 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-insights-runtime-extractor-tls podName:8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d nodeName:}" failed. No retries permitted until 2026-07-10 16:40:38.15918399 +0000 UTC m=+257.581060435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4bncj" (UID: "8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d") : secret "insights-runtime-extractor-tls" not found Jul 10 16:40:37.659454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.659300 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-data-volume\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:37.667819 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:37.667799 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9ttj\" (UniqueName: \"kubernetes.io/projected/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-api-access-h9ttj\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:38.163739 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:38.163691 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:38.163903 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:38.163780 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:38.163903 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:38.163808 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 16:40:38.163903 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:38.163879 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-rbac-proxy-cm podName:8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d nodeName:}" failed. No retries permitted until 2026-07-10 16:40:39.163856121 +0000 UTC m=+258.585732565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4bncj" (UID: "8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d") : configmap "kube-rbac-proxy" not found Jul 10 16:40:38.163903 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:38.163881 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 16:40:38.164055 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:38.163929 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-insights-runtime-extractor-tls podName:8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d nodeName:}" failed. No retries permitted until 2026-07-10 16:40:39.163918515 +0000 UTC m=+258.585794960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4bncj" (UID: "8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d") : secret "insights-runtime-extractor-tls" not found Jul 10 16:40:39.173399 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:39.173358 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:39.173880 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:39.173425 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:39.173880 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:39.173482 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 16:40:39.173880 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:39.173528 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 16:40:39.173880 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:39.173555 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-rbac-proxy-cm podName:8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d nodeName:}" failed. No retries permitted until 2026-07-10 16:40:41.17354 +0000 UTC m=+260.595416445 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4bncj" (UID: "8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d") : configmap "kube-rbac-proxy" not found Jul 10 16:40:39.173880 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:39.173570 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-insights-runtime-extractor-tls podName:8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d nodeName:}" failed. No retries permitted until 2026-07-10 16:40:41.173563525 +0000 UTC m=+260.595439970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4bncj" (UID: "8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d") : secret "insights-runtime-extractor-tls" not found Jul 10 16:40:39.979195 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:39.979165 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-metrics-certs\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:39.979347 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:39.979214 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:39.979347 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:39.979301 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 16:40:39.979432 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:39.979352 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-metrics-certs podName:6a5712a8-cdf6-4acc-afb0-21827ce7bbf4 nodeName:}" failed. No retries permitted until 2026-07-10 16:40:47.979339598 +0000 UTC m=+267.401216042 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-metrics-certs") pod "router-default-7b9b6c86cb-knlg6" (UID: "6a5712a8-cdf6-4acc-afb0-21827ce7bbf4") : secret "router-metrics-certs-default" not found Jul 10 16:40:39.979432 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:39.979374 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle podName:6a5712a8-cdf6-4acc-afb0-21827ce7bbf4 nodeName:}" failed. No retries permitted until 2026-07-10 16:40:47.979363847 +0000 UTC m=+267.401240292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle") pod "router-default-7b9b6c86cb-knlg6" (UID: "6a5712a8-cdf6-4acc-afb0-21827ce7bbf4") : configmap references non-existent config key: service-ca.crt Jul 10 16:40:39.979432 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:39.979309 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:39.979432 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:39.979423 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 16:40:39.979569 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:39.979439 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d74bffcbd-cvthk: secret "image-registry-tls" not found Jul 10 16:40:39.979569 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:39.979474 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls podName:2411ecc8-9b66-4410-a942-74fd98224442 nodeName:}" failed. No retries permitted until 2026-07-10 16:40:47.979464064 +0000 UTC m=+267.401340509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls") pod "image-registry-7d74bffcbd-cvthk" (UID: "2411ecc8-9b66-4410-a942-74fd98224442") : secret "image-registry-tls" not found Jul 10 16:40:40.481471 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:40.481445 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m7vs8_2d492a8c-19e2-462a-85bc-ac78c1898d73/dns-node-resolver/0.log" Jul 10 16:40:41.188313 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:41.188269 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:41.188522 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:41.188353 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:41.188522 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:41.188391 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 16:40:41.188522 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:41.188451 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-rbac-proxy-cm podName:8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d nodeName:}" failed. No retries permitted until 2026-07-10 16:40:45.188436838 +0000 UTC m=+264.610313288 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4bncj" (UID: "8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d") : configmap "kube-rbac-proxy" not found Jul 10 16:40:41.188522 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:41.188512 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 16:40:41.188700 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:41.188584 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-insights-runtime-extractor-tls podName:8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d nodeName:}" failed. No retries permitted until 2026-07-10 16:40:45.188565786 +0000 UTC m=+264.610442233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4bncj" (UID: "8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d") : secret "insights-runtime-extractor-tls" not found Jul 10 16:40:41.282162 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:41.282139 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dsv6h_1d54c0b4-7530-45fb-804c-777c15ff35eb/node-ca/0.log" Jul 10 16:40:42.082519 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:42.082490 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hbr2f_7ea783bf-0092-4036-a31b-bc4111a252c5/serve-healthcheck-canary/0.log" Jul 10 16:40:42.883485 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:42.883418 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-jf7sf_f6f755a9-cc53-4a26-a3ea-8054a806bf74/kube-storage-version-migrator-operator/0.log" Jul 10 16:40:45.221792 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:45.221758 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:45.222150 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:45.221825 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:45.222150 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:45.221891 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 16:40:45.222150 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:45.221898 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 16:40:45.222150 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:45.221941 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-rbac-proxy-cm podName:8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d nodeName:}" failed. No retries permitted until 2026-07-10 16:40:53.221928814 +0000 UTC m=+272.643805259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4bncj" (UID: "8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d") : configmap "kube-rbac-proxy" not found Jul 10 16:40:45.222150 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:45.221953 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-insights-runtime-extractor-tls podName:8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d nodeName:}" failed. No retries permitted until 2026-07-10 16:40:53.221948084 +0000 UTC m=+272.643824529 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4bncj" (UID: "8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d") : secret "insights-runtime-extractor-tls" not found Jul 10 16:40:48.044421 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:48.044377 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:48.044421 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:48.044427 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:48.044874 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:48.044490 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-metrics-certs\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:48.044874 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:40:48.044630 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle podName:6a5712a8-cdf6-4acc-afb0-21827ce7bbf4 nodeName:}" failed. No retries permitted until 2026-07-10 16:41:04.044610932 +0000 UTC m=+283.466487389 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle") pod "router-default-7b9b6c86cb-knlg6" (UID: "6a5712a8-cdf6-4acc-afb0-21827ce7bbf4") : configmap references non-existent config key: service-ca.crt Jul 10 16:40:48.046701 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:48.046675 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-metrics-certs\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:40:48.046814 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:48.046709 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls\") pod \"image-registry-7d74bffcbd-cvthk\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:48.161185 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:48.161161 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:48.278836 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:48.278805 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d74bffcbd-cvthk"] Jul 10 16:40:48.281598 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:40:48.281576 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2411ecc8_9b66_4410_a942_74fd98224442.slice/crio-e0d4e91b0ad48329cfd05819de2b0596e5882ffcac580cadaed6ee5209d07df9 WatchSource:0}: Error finding container e0d4e91b0ad48329cfd05819de2b0596e5882ffcac580cadaed6ee5209d07df9: Status 404 returned error can't find the container with id e0d4e91b0ad48329cfd05819de2b0596e5882ffcac580cadaed6ee5209d07df9 Jul 10 16:40:49.076487 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:49.076452 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" event={"ID":"2411ecc8-9b66-4410-a942-74fd98224442","Type":"ContainerStarted","Data":"2182956b61eeb7640cf460adec8ba44e8a16149caba3ffff5ee176b09a08ae23"} Jul 10 16:40:49.076914 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:49.076529 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" event={"ID":"2411ecc8-9b66-4410-a942-74fd98224442","Type":"ContainerStarted","Data":"e0d4e91b0ad48329cfd05819de2b0596e5882ffcac580cadaed6ee5209d07df9"} Jul 10 16:40:49.076914 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:49.076588 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:40:49.095109 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:49.094992 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" podStartSLOduration=17.094977046 podStartE2EDuration="17.094977046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:40:49.094675568 +0000 UTC m=+268.516552035" watchObservedRunningTime="2026-07-10 16:40:49.094977046 +0000 UTC m=+268.516853494" Jul 10 16:40:53.285181 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:53.285148 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:53.285620 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:53.285210 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:53.285681 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:53.285634 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:53.287468 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:53.287445 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bncj\" (UID: \"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d\") " pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:53.419785 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:53.419763 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4bncj" Jul 10 16:40:53.540896 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:53.540832 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4bncj"] Jul 10 16:40:53.543950 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:40:53.543918 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9f9cd0_98a6_4deb_8032_e0c4f9bbbc9d.slice/crio-831449c2ef38a379d21e6f8ba34c77c7f1df9d7e936b2bbaa752d952534158ad WatchSource:0}: Error finding container 831449c2ef38a379d21e6f8ba34c77c7f1df9d7e936b2bbaa752d952534158ad: Status 404 returned error can't find the container with id 831449c2ef38a379d21e6f8ba34c77c7f1df9d7e936b2bbaa752d952534158ad Jul 10 16:40:54.092515 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:54.092486 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4bncj" event={"ID":"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d","Type":"ContainerStarted","Data":"831449c2ef38a379d21e6f8ba34c77c7f1df9d7e936b2bbaa752d952534158ad"} Jul 10 16:40:58.106018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:58.105990 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4bncj" event={"ID":"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d","Type":"ContainerStarted","Data":"fece219182111c2fa97f6904a4a2c3851d6766e33644e911ee1f19752c0e482d"} Jul 10 16:40:58.106309 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:58.106042 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4bncj" event={"ID":"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d","Type":"ContainerStarted","Data":"8f06d9f8c4897d5b2dfac6fcf5dc07e6abffbe11bde730df605a3e4ba3dce040"} Jul 10 16:40:59.840606 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:59.840571 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d74bffcbd-cvthk"] Jul 10 16:40:59.896782 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:59.896736 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77d699fcbb-vqkjx"] Jul 10 16:40:59.903385 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:59.903358 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:40:59.919702 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:40:59.919679 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77d699fcbb-vqkjx"] Jul 10 16:41:00.035496 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.035454 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a725bedd-e82d-4abd-b3f6-888d01da0fc4-registry-tls\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.035496 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.035495 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a725bedd-e82d-4abd-b3f6-888d01da0fc4-registry-certificates\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.035716 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.035578 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzf52\" (UniqueName: \"kubernetes.io/projected/a725bedd-e82d-4abd-b3f6-888d01da0fc4-kube-api-access-rzf52\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.035716 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.035636 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a725bedd-e82d-4abd-b3f6-888d01da0fc4-trusted-ca\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.035842 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.035713 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a725bedd-e82d-4abd-b3f6-888d01da0fc4-installation-pull-secrets\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.035842 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.035780 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a725bedd-e82d-4abd-b3f6-888d01da0fc4-image-registry-private-configuration\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.035842 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.035812 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a725bedd-e82d-4abd-b3f6-888d01da0fc4-bound-sa-token\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.035842 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.035838 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a725bedd-e82d-4abd-b3f6-888d01da0fc4-ca-trust-extracted\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.137014 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.136981 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a725bedd-e82d-4abd-b3f6-888d01da0fc4-image-registry-private-configuration\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.137195 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.137025 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a725bedd-e82d-4abd-b3f6-888d01da0fc4-bound-sa-token\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.137195 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.137054 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a725bedd-e82d-4abd-b3f6-888d01da0fc4-ca-trust-extracted\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.137195 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.137095 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a725bedd-e82d-4abd-b3f6-888d01da0fc4-registry-tls\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.137195 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.137119 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a725bedd-e82d-4abd-b3f6-888d01da0fc4-registry-certificates\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.137195 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.137149 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzf52\" (UniqueName: \"kubernetes.io/projected/a725bedd-e82d-4abd-b3f6-888d01da0fc4-kube-api-access-rzf52\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.137195 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.137181 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a725bedd-e82d-4abd-b3f6-888d01da0fc4-trusted-ca\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.137502 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.137256 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a725bedd-e82d-4abd-b3f6-888d01da0fc4-installation-pull-secrets\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.137502 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.137456 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a725bedd-e82d-4abd-b3f6-888d01da0fc4-ca-trust-extracted\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.138212 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.138175 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a725bedd-e82d-4abd-b3f6-888d01da0fc4-registry-certificates\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.138523 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.138474 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a725bedd-e82d-4abd-b3f6-888d01da0fc4-trusted-ca\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.140029 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.139995 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a725bedd-e82d-4abd-b3f6-888d01da0fc4-image-registry-private-configuration\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.140163 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.140073 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a725bedd-e82d-4abd-b3f6-888d01da0fc4-installation-pull-secrets\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.140264 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.140238 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a725bedd-e82d-4abd-b3f6-888d01da0fc4-registry-tls\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.146807 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.146727 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a725bedd-e82d-4abd-b3f6-888d01da0fc4-bound-sa-token\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.146924 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.146908 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzf52\" (UniqueName: \"kubernetes.io/projected/a725bedd-e82d-4abd-b3f6-888d01da0fc4-kube-api-access-rzf52\") pod \"image-registry-77d699fcbb-vqkjx\" (UID: \"a725bedd-e82d-4abd-b3f6-888d01da0fc4\") " pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.218047 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.218008 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:00.673290 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:00.673269 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77d699fcbb-vqkjx"] Jul 10 16:41:00.675214 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:41:00.675194 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda725bedd_e82d_4abd_b3f6_888d01da0fc4.slice/crio-2b693c214b25776d6914e3aafc2f5a3379d4bdae8fe533882571728dfcc86b37 WatchSource:0}: Error finding container 2b693c214b25776d6914e3aafc2f5a3379d4bdae8fe533882571728dfcc86b37: Status 404 returned error can't find the container with id 2b693c214b25776d6914e3aafc2f5a3379d4bdae8fe533882571728dfcc86b37 Jul 10 16:41:01.118183 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:01.118143 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4bncj" event={"ID":"8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d","Type":"ContainerStarted","Data":"a46cab2380e7d93401bb23aae5ba808b78096de3654531f621d9e73e7c9415e4"} Jul 10 16:41:01.119493 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:01.119463 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" event={"ID":"a725bedd-e82d-4abd-b3f6-888d01da0fc4","Type":"ContainerStarted","Data":"4789dd363a475d3f2411ef7389a3672573da589d73bb69a610c3382809a10d03"} Jul 10 16:41:01.119593 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:01.119513 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" event={"ID":"a725bedd-e82d-4abd-b3f6-888d01da0fc4","Type":"ContainerStarted","Data":"2b693c214b25776d6914e3aafc2f5a3379d4bdae8fe533882571728dfcc86b37"} Jul 10 16:41:01.119593 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:01.119535 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:01.136343 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:01.136303 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4bncj" podStartSLOduration=17.18937506 podStartE2EDuration="24.136285906s" totalImagesPullingTime="6.946910846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:40:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:41:01.134164001 +0000 UTC m=+280.556040466" watchObservedRunningTime="2026-07-10 16:41:01.136285906 +0000 UTC m=+280.558162373" Jul 10 16:41:01.152639 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:01.152597 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" podStartSLOduration=2.152583746 podStartE2EDuration="2.152583746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:41:01.152269127 +0000 UTC m=+280.574145628" watchObservedRunningTime="2026-07-10 16:41:01.152583746 +0000 UTC m=+280.574460213" Jul 10 16:41:03.802299 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:41:03.802258 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-kp9dv" podUID="32d1ff51-842a-4cb2-b179-ffe84520aad3" Jul 10 16:41:04.064618 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:04.064550 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:41:04.065145 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:04.065125 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5712a8-cdf6-4acc-afb0-21827ce7bbf4-service-ca-bundle\") pod \"router-default-7b9b6c86cb-knlg6\" (UID: \"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4\") " pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:41:04.067637 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:04.067613 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:41:04.129564 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:04.129536 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kp9dv" Jul 10 16:41:04.189936 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:04.189908 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7b9b6c86cb-knlg6"] Jul 10 16:41:04.192871 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:41:04.192845 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5712a8_cdf6_4acc_afb0_21827ce7bbf4.slice/crio-5ce741438a047b695e76eabfe84abfe789d7ab2c2e45ff5551156e2fe55891e8 WatchSource:0}: Error finding container 5ce741438a047b695e76eabfe84abfe789d7ab2c2e45ff5551156e2fe55891e8: Status 404 returned error can't find the container with id 5ce741438a047b695e76eabfe84abfe789d7ab2c2e45ff5551156e2fe55891e8 Jul 10 16:41:05.133716 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:05.133685 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" event={"ID":"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4","Type":"ContainerStarted","Data":"5c08bd0695a5f87e6579c64dacf7918202f79ead65fd4d747347064179ce1713"} Jul 10 16:41:05.134081 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:05.133729 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" event={"ID":"6a5712a8-cdf6-4acc-afb0-21827ce7bbf4","Type":"ContainerStarted","Data":"5ce741438a047b695e76eabfe84abfe789d7ab2c2e45ff5551156e2fe55891e8"} Jul 10 16:41:05.158265 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:05.158220 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" podStartSLOduration=33.158205242 podStartE2EDuration="33.158205242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:41:05.157322436 +0000 UTC m=+284.579198924" watchObservedRunningTime="2026-07-10 16:41:05.158205242 +0000 UTC m=+284.580081708" Jul 10 16:41:06.068144 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:06.068111 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:41:06.070435 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:06.070412 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:41:06.137008 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:06.136984 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:41:06.138058 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:06.138042 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7b9b6c86cb-knlg6" Jul 10 16:41:07.185415 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:07.185383 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:41:07.187490 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:07.187473 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32d1ff51-842a-4cb2-b179-ffe84520aad3-metrics-tls\") pod \"dns-default-kp9dv\" (UID: \"32d1ff51-842a-4cb2-b179-ffe84520aad3\") " pod="openshift-dns/dns-default-kp9dv" Jul 10 16:41:07.432730 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:07.432703 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jqrx8\"" Jul 10 16:41:07.441085 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:07.441033 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kp9dv" Jul 10 16:41:07.562601 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:07.562575 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kp9dv"] Jul 10 16:41:07.565844 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:41:07.565812 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d1ff51_842a_4cb2_b179_ffe84520aad3.slice/crio-0c34707dfa533df5b70347dc2e3a9b6a72dc224926a438ff1e49d9f3141ffd06 WatchSource:0}: Error finding container 0c34707dfa533df5b70347dc2e3a9b6a72dc224926a438ff1e49d9f3141ffd06: Status 404 returned error can't find the container with id 0c34707dfa533df5b70347dc2e3a9b6a72dc224926a438ff1e49d9f3141ffd06 Jul 10 16:41:07.703367 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:07.703312 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-qr7rl"] Jul 10 16:41:07.707512 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:07.707497 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-qr7rl" Jul 10 16:41:07.710121 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:07.710102 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 10 16:41:07.710212 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:07.710148 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-swtmt\"" Jul 10 16:41:07.713931 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:07.713909 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-qr7rl"] Jul 10 16:41:07.790223 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:07.790199 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d394d15a-75b8-4f95-9bd3-06aaa8b9d6c3-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-qr7rl\" (UID: \"d394d15a-75b8-4f95-9bd3-06aaa8b9d6c3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-qr7rl" Jul 10 16:41:07.890905 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:07.890877 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d394d15a-75b8-4f95-9bd3-06aaa8b9d6c3-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-qr7rl\" (UID: \"d394d15a-75b8-4f95-9bd3-06aaa8b9d6c3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-qr7rl" Jul 10 16:41:07.893104 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:07.893084 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d394d15a-75b8-4f95-9bd3-06aaa8b9d6c3-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-qr7rl\" (UID: \"d394d15a-75b8-4f95-9bd3-06aaa8b9d6c3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-qr7rl" Jul 10 16:41:08.017683 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:08.017624 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-qr7rl" Jul 10 16:41:08.133834 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:08.133803 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-qr7rl"] Jul 10 16:41:08.136443 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:41:08.136419 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd394d15a_75b8_4f95_9bd3_06aaa8b9d6c3.slice/crio-d8730621e48aeb1a8aab82f7f743ff06ca9e2ff684a3f7ed14227ffa6dd67c07 WatchSource:0}: Error finding container d8730621e48aeb1a8aab82f7f743ff06ca9e2ff684a3f7ed14227ffa6dd67c07: Status 404 returned error can't find the container with id d8730621e48aeb1a8aab82f7f743ff06ca9e2ff684a3f7ed14227ffa6dd67c07 Jul 10 16:41:08.143659 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:08.143634 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kp9dv" event={"ID":"32d1ff51-842a-4cb2-b179-ffe84520aad3","Type":"ContainerStarted","Data":"0c34707dfa533df5b70347dc2e3a9b6a72dc224926a438ff1e49d9f3141ffd06"} Jul 10 16:41:08.144645 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:08.144626 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-qr7rl" event={"ID":"d394d15a-75b8-4f95-9bd3-06aaa8b9d6c3","Type":"ContainerStarted","Data":"d8730621e48aeb1a8aab82f7f743ff06ca9e2ff684a3f7ed14227ffa6dd67c07"} Jul 10 16:41:09.846339 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:09.846268 2585 patch_prober.go:28] interesting pod/image-registry-7d74bffcbd-cvthk container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 16:41:09.846339 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:09.846318 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" podUID="2411ecc8-9b66-4410-a942-74fd98224442" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 16:41:10.152561 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:10.152529 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-qr7rl" event={"ID":"d394d15a-75b8-4f95-9bd3-06aaa8b9d6c3","Type":"ContainerStarted","Data":"949b641d73d636035fbf09231f42e95e5e0886ab67d34c64b8c4c0aaf2cda92a"} Jul 10 16:41:10.152717 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:10.152586 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-qr7rl" Jul 10 16:41:10.154328 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:10.154302 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kp9dv" event={"ID":"32d1ff51-842a-4cb2-b179-ffe84520aad3","Type":"ContainerStarted","Data":"efe7d5667e2395c49c7be20993ce0aa13b00bac257405f594ecc8a51fc10d3b5"} Jul 10 16:41:10.154445 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:10.154338 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kp9dv" event={"ID":"32d1ff51-842a-4cb2-b179-ffe84520aad3","Type":"ContainerStarted","Data":"b3ab665fb84654310e20d5bff4763849412ebf4016f5db2430b6ca94fdd355db"} Jul 10 16:41:10.154445 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:10.154434 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kp9dv" Jul 10 16:41:10.157137 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:10.157119 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-qr7rl" Jul 10 16:41:10.168241 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:10.168204 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-qr7rl" podStartSLOduration=1.7505268090000001 podStartE2EDuration="3.16819293s" totalImagesPullingTime="1.417666121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:41:10.167607809 +0000 UTC m=+289.589484288" watchObservedRunningTime="2026-07-10 16:41:10.16819293 +0000 UTC m=+289.590069396" Jul 10 16:41:10.199709 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:10.199667 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kp9dv" podStartSLOduration=251.21099048 podStartE2EDuration="4m13.199651669s" totalImagesPullingTime="1.988661189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:36:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:41:10.199302925 +0000 UTC m=+289.621179411" watchObservedRunningTime="2026-07-10 16:41:10.199651669 +0000 UTC m=+289.621528136" Jul 10 16:41:13.331266 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.331233 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6ff59fbb78-m4ccl"] Jul 10 16:41:13.338298 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.338275 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.341019 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.340998 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 16:41:13.342337 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.342313 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 16:41:13.342462 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.342356 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 16:41:13.342462 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.342422 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 16:41:13.342579 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.342499 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 16:41:13.342579 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.342425 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 16:41:13.342863 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.342336 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 16:41:13.342863 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.342703 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-b49fg\"" Jul 10 16:41:13.344449 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.344423 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ff59fbb78-m4ccl"] Jul 10 16:41:13.347475 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.347457 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 16:41:13.429060 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.429029 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-oauth-serving-cert\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.429159 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.429076 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-service-ca\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.429159 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.429124 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-console-config\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.429273 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.429158 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-console-serving-cert\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.429273 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.429178 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-trusted-ca-bundle\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.429273 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.429243 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-console-oauth-config\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.429387 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.429275 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzw4q\" (UniqueName: \"kubernetes.io/projected/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-kube-api-access-xzw4q\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.529565 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.529537 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-console-config\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.529565 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.529564 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-console-serving-cert\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.529729 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.529579 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-trusted-ca-bundle\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.529729 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.529602 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-console-oauth-config\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.529729 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.529619 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzw4q\" (UniqueName: \"kubernetes.io/projected/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-kube-api-access-xzw4q\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.529904 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.529807 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-oauth-serving-cert\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.529904 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.529863 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-service-ca\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.530344 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.530320 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-console-config\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.530509 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.530492 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-service-ca\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.530575 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.530506 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-oauth-serving-cert\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.530575 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.530536 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-trusted-ca-bundle\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.531994 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.531964 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-console-oauth-config\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.532126 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.532109 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-console-serving-cert\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.537550 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.537528 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzw4q\" (UniqueName: \"kubernetes.io/projected/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-kube-api-access-xzw4q\") pod \"console-6ff59fbb78-m4ccl\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.647993 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.647973 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:13.771098 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:13.771071 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ff59fbb78-m4ccl"] Jul 10 16:41:13.774231 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:41:13.774206 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a42eb87_0ac1_4d9f_a7f7_2822001805e3.slice/crio-a2ca38129cd1c9b69d7615ec799b4a644394a38891698f66e5b1310bfdfa6ac8 WatchSource:0}: Error finding container a2ca38129cd1c9b69d7615ec799b4a644394a38891698f66e5b1310bfdfa6ac8: Status 404 returned error can't find the container with id a2ca38129cd1c9b69d7615ec799b4a644394a38891698f66e5b1310bfdfa6ac8 Jul 10 16:41:14.170529 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:14.170497 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff59fbb78-m4ccl" event={"ID":"2a42eb87-0ac1-4d9f-a7f7-2822001805e3","Type":"ContainerStarted","Data":"a2ca38129cd1c9b69d7615ec799b4a644394a38891698f66e5b1310bfdfa6ac8"} Jul 10 16:41:17.098491 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.098460 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx"] Jul 10 16:41:17.101570 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.101554 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" Jul 10 16:41:17.107128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.107096 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 16:41:17.107823 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.107728 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-nkpt6"] Jul 10 16:41:17.107946 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.107829 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 10 16:41:17.107946 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.107844 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 16:41:17.107946 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.107849 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 10 16:41:17.107946 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.107865 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 16:41:17.108137 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.108123 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wnxpx\"" Jul 10 16:41:17.111023 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.111006 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.113650 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.113606 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 10 16:41:17.113934 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.113910 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 10 16:41:17.114044 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.113910 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-n4q2g\"" Jul 10 16:41:17.114978 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.114960 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 10 16:41:17.116286 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.116268 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx"] Jul 10 16:41:17.121094 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.121076 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6tmzw"] Jul 10 16:41:17.124353 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.124336 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-nkpt6"] Jul 10 16:41:17.124434 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.124424 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.127505 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.127488 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 16:41:17.127585 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.127519 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 16:41:17.127585 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.127548 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 16:41:17.127585 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.127490 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5l4q2\"" Jul 10 16:41:17.181511 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.181487 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff59fbb78-m4ccl" event={"ID":"2a42eb87-0ac1-4d9f-a7f7-2822001805e3","Type":"ContainerStarted","Data":"dca10cad899bbbe9fff44dbbc68bb29b20d9ab05b5414627dc55c0c905c4638f"} Jul 10 16:41:17.203624 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.203584 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6ff59fbb78-m4ccl" podStartSLOduration=1.4977017369999999 podStartE2EDuration="4.203571903s" totalImagesPullingTime="2.705870166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:41:17.203304505 +0000 UTC m=+296.625180969" watchObservedRunningTime="2026-07-10 16:41:17.203571903 +0000 UTC m=+296.625448369" Jul 10 16:41:17.262666 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.262637 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-sys\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.262777 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.262675 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73e50952-6b77-4e60-a941-91629e495158-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-n6hcx\" (UID: \"73e50952-6b77-4e60-a941-91629e495158\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" Jul 10 16:41:17.262777 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.262694 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73e50952-6b77-4e60-a941-91629e495158-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-n6hcx\" (UID: \"73e50952-6b77-4e60-a941-91629e495158\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" Jul 10 16:41:17.262777 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.262740 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f425e231-4c8e-4482-b2cc-bec2266b9a93-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-nkpt6\" (UID: \"f425e231-4c8e-4482-b2cc-bec2266b9a93\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.262882 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.262786 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.262882 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.262850 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f425e231-4c8e-4482-b2cc-bec2266b9a93-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-nkpt6\" (UID: \"f425e231-4c8e-4482-b2cc-bec2266b9a93\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.262943 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.262887 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-node-exporter-accelerators-collector-config\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.262943 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.262910 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-root\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.262943 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.262928 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-node-exporter-tls\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.263034 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.262995 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-node-exporter-textfile\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.263034 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.263020 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dkzf\" (UniqueName: \"kubernetes.io/projected/73e50952-6b77-4e60-a941-91629e495158-kube-api-access-2dkzf\") pod \"openshift-state-metrics-84bd68b896-n6hcx\" (UID: \"73e50952-6b77-4e60-a941-91629e495158\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" Jul 10 16:41:17.263092 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.263059 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-node-exporter-wtmp\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.263092 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.263076 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73e50952-6b77-4e60-a941-91629e495158-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-n6hcx\" (UID: \"73e50952-6b77-4e60-a941-91629e495158\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" Jul 10 16:41:17.263156 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.263092 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f425e231-4c8e-4482-b2cc-bec2266b9a93-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-nkpt6\" (UID: \"f425e231-4c8e-4482-b2cc-bec2266b9a93\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.263156 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.263123 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-metrics-client-ca\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.263156 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.263140 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vj5k\" (UniqueName: \"kubernetes.io/projected/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-kube-api-access-7vj5k\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.263240 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.263176 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j9pw\" (UniqueName: \"kubernetes.io/projected/f425e231-4c8e-4482-b2cc-bec2266b9a93-kube-api-access-4j9pw\") pod \"kube-state-metrics-785778ff74-nkpt6\" (UID: \"f425e231-4c8e-4482-b2cc-bec2266b9a93\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.263240 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.263201 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f425e231-4c8e-4482-b2cc-bec2266b9a93-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-nkpt6\" (UID: \"f425e231-4c8e-4482-b2cc-bec2266b9a93\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.263240 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.263218 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f425e231-4c8e-4482-b2cc-bec2266b9a93-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-nkpt6\" (UID: \"f425e231-4c8e-4482-b2cc-bec2266b9a93\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.364112 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.364091 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-sys\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.364212 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.364121 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73e50952-6b77-4e60-a941-91629e495158-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-n6hcx\" (UID: \"73e50952-6b77-4e60-a941-91629e495158\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" Jul 10 16:41:17.364212 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.364138 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73e50952-6b77-4e60-a941-91629e495158-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-n6hcx\" (UID: \"73e50952-6b77-4e60-a941-91629e495158\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" Jul 10 16:41:17.364294 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.364205 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-sys\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.364294 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.364255 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f425e231-4c8e-4482-b2cc-bec2266b9a93-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-nkpt6\" (UID: \"f425e231-4c8e-4482-b2cc-bec2266b9a93\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.364294 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.364285 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.364402 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.364330 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f425e231-4c8e-4482-b2cc-bec2266b9a93-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-nkpt6\" (UID: \"f425e231-4c8e-4482-b2cc-bec2266b9a93\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.364402 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.364381 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-node-exporter-accelerators-collector-config\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.364502 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.364409 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-root\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.364502 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.364476 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-root\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.364904 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.364880 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-node-exporter-tls\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.365092 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:41:17.365074 2585 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 16:41:17.365264 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:41:17.365250 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-node-exporter-tls podName:5eecaa1e-89cc-4bb3-97bf-05d9d5d32683 nodeName:}" failed. No retries permitted until 2026-07-10 16:41:17.865224546 +0000 UTC m=+297.287100991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-node-exporter-tls") pod "node-exporter-6tmzw" (UID: "5eecaa1e-89cc-4bb3-97bf-05d9d5d32683") : secret "node-exporter-tls" not found Jul 10 16:41:17.365398 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.365283 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-node-exporter-accelerators-collector-config\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.365529 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.365503 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-node-exporter-textfile\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.365646 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.365546 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dkzf\" (UniqueName: \"kubernetes.io/projected/73e50952-6b77-4e60-a941-91629e495158-kube-api-access-2dkzf\") pod \"openshift-state-metrics-84bd68b896-n6hcx\" (UID: \"73e50952-6b77-4e60-a941-91629e495158\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" Jul 10 16:41:17.365646 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.365581 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-node-exporter-wtmp\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.365646 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.365607 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73e50952-6b77-4e60-a941-91629e495158-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-n6hcx\" (UID: \"73e50952-6b77-4e60-a941-91629e495158\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" Jul 10 16:41:17.365835 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.365654 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f425e231-4c8e-4482-b2cc-bec2266b9a93-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-nkpt6\" (UID: \"f425e231-4c8e-4482-b2cc-bec2266b9a93\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.365835 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.365720 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-metrics-client-ca\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.365835 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.365770 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vj5k\" (UniqueName: \"kubernetes.io/projected/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-kube-api-access-7vj5k\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.365835 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.365793 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-node-exporter-textfile\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.365835 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.365802 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j9pw\" (UniqueName: \"kubernetes.io/projected/f425e231-4c8e-4482-b2cc-bec2266b9a93-kube-api-access-4j9pw\") pod \"kube-state-metrics-785778ff74-nkpt6\" (UID: \"f425e231-4c8e-4482-b2cc-bec2266b9a93\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.366080 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.365845 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f425e231-4c8e-4482-b2cc-bec2266b9a93-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-nkpt6\" (UID: \"f425e231-4c8e-4482-b2cc-bec2266b9a93\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.366080 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.365877 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f425e231-4c8e-4482-b2cc-bec2266b9a93-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-nkpt6\" (UID: \"f425e231-4c8e-4482-b2cc-bec2266b9a93\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.367212 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.366452 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f425e231-4c8e-4482-b2cc-bec2266b9a93-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-nkpt6\" (UID: \"f425e231-4c8e-4482-b2cc-bec2266b9a93\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.367212 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.366575 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f425e231-4c8e-4482-b2cc-bec2266b9a93-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-nkpt6\" (UID: \"f425e231-4c8e-4482-b2cc-bec2266b9a93\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.367212 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.366884 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f425e231-4c8e-4482-b2cc-bec2266b9a93-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-nkpt6\" (UID: \"f425e231-4c8e-4482-b2cc-bec2266b9a93\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.367212 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.366941 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-node-exporter-wtmp\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.367485 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.367215 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.367485 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.367267 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-metrics-client-ca\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.367485 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.367327 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73e50952-6b77-4e60-a941-91629e495158-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-n6hcx\" (UID: \"73e50952-6b77-4e60-a941-91629e495158\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" Jul 10 16:41:17.367656 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.367629 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73e50952-6b77-4e60-a941-91629e495158-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-n6hcx\" (UID: \"73e50952-6b77-4e60-a941-91629e495158\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" Jul 10 16:41:17.367734 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.367716 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f425e231-4c8e-4482-b2cc-bec2266b9a93-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-nkpt6\" (UID: \"f425e231-4c8e-4482-b2cc-bec2266b9a93\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.367946 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.367927 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f425e231-4c8e-4482-b2cc-bec2266b9a93-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-nkpt6\" (UID: \"f425e231-4c8e-4482-b2cc-bec2266b9a93\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.368066 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.368046 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73e50952-6b77-4e60-a941-91629e495158-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-n6hcx\" (UID: \"73e50952-6b77-4e60-a941-91629e495158\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" Jul 10 16:41:17.379482 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.379451 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vj5k\" (UniqueName: \"kubernetes.io/projected/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-kube-api-access-7vj5k\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.380279 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.380260 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dkzf\" (UniqueName: \"kubernetes.io/projected/73e50952-6b77-4e60-a941-91629e495158-kube-api-access-2dkzf\") pod \"openshift-state-metrics-84bd68b896-n6hcx\" (UID: \"73e50952-6b77-4e60-a941-91629e495158\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" Jul 10 16:41:17.380366 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.380344 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j9pw\" (UniqueName: \"kubernetes.io/projected/f425e231-4c8e-4482-b2cc-bec2266b9a93-kube-api-access-4j9pw\") pod \"kube-state-metrics-785778ff74-nkpt6\" (UID: \"f425e231-4c8e-4482-b2cc-bec2266b9a93\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.413545 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.413522 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" Jul 10 16:41:17.421682 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.421663 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" Jul 10 16:41:17.550392 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.550127 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx"] Jul 10 16:41:17.552633 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:41:17.552603 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e50952_6b77_4e60_a941_91629e495158.slice/crio-64c0166959d08f29a4a5803daf1c682b4a9612201edafcdb7b444a478b8d450e WatchSource:0}: Error finding container 64c0166959d08f29a4a5803daf1c682b4a9612201edafcdb7b444a478b8d450e: Status 404 returned error can't find the container with id 64c0166959d08f29a4a5803daf1c682b4a9612201edafcdb7b444a478b8d450e Jul 10 16:41:17.567807 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.567784 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-nkpt6"] Jul 10 16:41:17.570249 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:41:17.570228 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf425e231_4c8e_4482_b2cc_bec2266b9a93.slice/crio-160487c841847db6985334053ce2aeb1141d939eec96fb65fa56a8fb2ce96c60 WatchSource:0}: Error finding container 160487c841847db6985334053ce2aeb1141d939eec96fb65fa56a8fb2ce96c60: Status 404 returned error can't find the container with id 160487c841847db6985334053ce2aeb1141d939eec96fb65fa56a8fb2ce96c60 Jul 10 16:41:17.870615 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.870528 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-node-exporter-tls\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:17.872735 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:17.872711 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5eecaa1e-89cc-4bb3-97bf-05d9d5d32683-node-exporter-tls\") pod \"node-exporter-6tmzw\" (UID: \"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683\") " pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:18.033136 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:18.033110 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6tmzw" Jul 10 16:41:18.044700 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:41:18.044676 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eecaa1e_89cc_4bb3_97bf_05d9d5d32683.slice/crio-2b22fc39e7fd99000d51b57e9e50fb8891abaafd272f026c037ab1750c4fbfd9 WatchSource:0}: Error finding container 2b22fc39e7fd99000d51b57e9e50fb8891abaafd272f026c037ab1750c4fbfd9: Status 404 returned error can't find the container with id 2b22fc39e7fd99000d51b57e9e50fb8891abaafd272f026c037ab1750c4fbfd9 Jul 10 16:41:18.187637 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:18.187560 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6tmzw" event={"ID":"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683","Type":"ContainerStarted","Data":"2b22fc39e7fd99000d51b57e9e50fb8891abaafd272f026c037ab1750c4fbfd9"} Jul 10 16:41:18.189712 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:18.189677 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" event={"ID":"73e50952-6b77-4e60-a941-91629e495158","Type":"ContainerStarted","Data":"16ba78a426afbe4aebf68dcb048bdb7eaab7429cc617c003cb028f99989233cf"} Jul 10 16:41:18.189870 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:18.189762 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" event={"ID":"73e50952-6b77-4e60-a941-91629e495158","Type":"ContainerStarted","Data":"08904e94fe296df4846e0a370e0a017547bdfb6addbc84ddfb3fe0e827594d50"} Jul 10 16:41:18.189870 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:18.189797 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" event={"ID":"73e50952-6b77-4e60-a941-91629e495158","Type":"ContainerStarted","Data":"64c0166959d08f29a4a5803daf1c682b4a9612201edafcdb7b444a478b8d450e"} Jul 10 16:41:18.191002 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:18.190966 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" event={"ID":"f425e231-4c8e-4482-b2cc-bec2266b9a93","Type":"ContainerStarted","Data":"160487c841847db6985334053ce2aeb1141d939eec96fb65fa56a8fb2ce96c60"} Jul 10 16:41:19.197478 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:19.197438 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" event={"ID":"73e50952-6b77-4e60-a941-91629e495158","Type":"ContainerStarted","Data":"c117c73c169fd3c226ca003fa55a1d976a0719845e25e82406867e6fea6d13c1"} Jul 10 16:41:19.217880 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:19.217825 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n6hcx" podStartSLOduration=1.260552439 podStartE2EDuration="2.217805447s" totalImagesPullingTime="957.253008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:41:19.21650522 +0000 UTC m=+298.638381711" watchObservedRunningTime="2026-07-10 16:41:19.217805447 +0000 UTC m=+298.639681916" Jul 10 16:41:19.320944 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:41:19.320902 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6c49f" podUID="08a4d929-2c36-40ac-a3e6-8c5caf0c1788" Jul 10 16:41:19.846357 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:19.845937 2585 patch_prober.go:28] interesting pod/image-registry-7d74bffcbd-cvthk container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 16:41:19.846357 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:19.846002 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" podUID="2411ecc8-9b66-4410-a942-74fd98224442" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 16:41:20.161150 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:20.161121 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kp9dv" Jul 10 16:41:20.204007 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:20.203974 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" event={"ID":"f425e231-4c8e-4482-b2cc-bec2266b9a93","Type":"ContainerStarted","Data":"a6588afc6952fde8edbaf8bf717ad843d18baebf6eee1e6ca47bd72f47cccf61"} Jul 10 16:41:20.204436 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:20.204036 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" event={"ID":"f425e231-4c8e-4482-b2cc-bec2266b9a93","Type":"ContainerStarted","Data":"03b35bb5551938d8159476592684ef37e0c664142417e7cb658121316d94dd08"} Jul 10 16:41:20.204436 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:20.204066 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" event={"ID":"f425e231-4c8e-4482-b2cc-bec2266b9a93","Type":"ContainerStarted","Data":"bcb8ace4a63f6948db1e880f2986ace1b7246c02729a652bfac93aa944dfd8ee"} Jul 10 16:41:20.205672 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:20.205640 2585 generic.go:356] "Generic (PLEG): container finished" podID="5eecaa1e-89cc-4bb3-97bf-05d9d5d32683" containerID="83850f145bd37e77c67b0e8af3cfb079f1a9daa9cc06181ebaf976e2bf655288" exitCode=0 Jul 10 16:41:20.205832 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:20.205725 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6tmzw" event={"ID":"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683","Type":"ContainerDied","Data":"83850f145bd37e77c67b0e8af3cfb079f1a9daa9cc06181ebaf976e2bf655288"} Jul 10 16:41:20.223228 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:20.223183 2585 patch_prober.go:28] interesting pod/image-registry-77d699fcbb-vqkjx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 16:41:20.223350 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:20.223228 2585 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" podUID="a725bedd-e82d-4abd-b3f6-888d01da0fc4" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 16:41:20.223780 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:20.223710 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-nkpt6" podStartSLOduration=1.349091005 podStartE2EDuration="3.223693314s" totalImagesPullingTime="1.874602309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:41:20.222805941 +0000 UTC m=+299.644682410" watchObservedRunningTime="2026-07-10 16:41:20.223693314 +0000 UTC m=+299.645569783" Jul 10 16:41:20.951600 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:20.951575 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 16:41:20.951780 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:20.951639 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 16:41:20.957778 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:20.957759 2585 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 16:41:21.210964 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.210891 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6tmzw" event={"ID":"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683","Type":"ContainerStarted","Data":"1c72a592f85256105cd0a8384f5fed4cc5c11c23d53ba1ac8c573666142a37bb"} Jul 10 16:41:21.210964 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.210950 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6tmzw" event={"ID":"5eecaa1e-89cc-4bb3-97bf-05d9d5d32683","Type":"ContainerStarted","Data":"df4f7d3e7db65ac7232f74606c838cd4837ebdc7a826a92a526e5221c1a26bd6"} Jul 10 16:41:21.232443 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.232406 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6tmzw" podStartSLOduration=2.758859048 podStartE2EDuration="4.232391505s" totalImagesPullingTime="1.397921977s" totalInitContainerRuntime="75.61048ms" isStatefulPod=true podCreationTimestamp="2026-07-10 16:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:41:21.232231786 +0000 UTC m=+300.654108257" watchObservedRunningTime="2026-07-10 16:41:21.232391505 +0000 UTC m=+300.654267971" Jul 10 16:41:21.402979 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.402956 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-d7c4f5f87-jx56r"] Jul 10 16:41:21.405797 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.405781 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.408642 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.408618 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 10 16:41:21.408784 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.408624 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 10 16:41:21.408784 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.408659 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 10 16:41:21.408784 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.408708 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cm6351c5vcekl\"" Jul 10 16:41:21.408784 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.408660 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 10 16:41:21.408784 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.408644 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bz8cv\"" Jul 10 16:41:21.416537 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.416519 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d7c4f5f87-jx56r"] Jul 10 16:41:21.601800 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.601699 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/60146d46-3d75-41de-8d5a-3dba67f81a88-audit-log\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.601800 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.601732 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/60146d46-3d75-41de-8d5a-3dba67f81a88-client-ca-bundle\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.601968 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.601803 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/60146d46-3d75-41de-8d5a-3dba67f81a88-secret-metrics-server-tls\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.601968 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.601826 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60146d46-3d75-41de-8d5a-3dba67f81a88-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.601968 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.601896 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/60146d46-3d75-41de-8d5a-3dba67f81a88-metrics-server-audit-profiles\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.601968 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.601931 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjzmn\" (UniqueName: \"kubernetes.io/projected/60146d46-3d75-41de-8d5a-3dba67f81a88-kube-api-access-gjzmn\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.602123 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.601980 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/60146d46-3d75-41de-8d5a-3dba67f81a88-secret-metrics-server-client-certs\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.703070 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.703042 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/60146d46-3d75-41de-8d5a-3dba67f81a88-secret-metrics-server-tls\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.703208 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.703080 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60146d46-3d75-41de-8d5a-3dba67f81a88-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.703208 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.703117 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/60146d46-3d75-41de-8d5a-3dba67f81a88-metrics-server-audit-profiles\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.703208 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.703149 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjzmn\" (UniqueName: \"kubernetes.io/projected/60146d46-3d75-41de-8d5a-3dba67f81a88-kube-api-access-gjzmn\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.703368 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.703208 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/60146d46-3d75-41de-8d5a-3dba67f81a88-secret-metrics-server-client-certs\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.703368 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.703253 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/60146d46-3d75-41de-8d5a-3dba67f81a88-audit-log\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.703368 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.703277 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/60146d46-3d75-41de-8d5a-3dba67f81a88-client-ca-bundle\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.703674 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.703648 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/60146d46-3d75-41de-8d5a-3dba67f81a88-audit-log\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.703927 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.703897 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60146d46-3d75-41de-8d5a-3dba67f81a88-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.704175 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.704154 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/60146d46-3d75-41de-8d5a-3dba67f81a88-metrics-server-audit-profiles\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.705569 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.705543 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/60146d46-3d75-41de-8d5a-3dba67f81a88-secret-metrics-server-tls\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.705653 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.705615 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/60146d46-3d75-41de-8d5a-3dba67f81a88-secret-metrics-server-client-certs\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.705776 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.705757 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/60146d46-3d75-41de-8d5a-3dba67f81a88-client-ca-bundle\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.710790 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.710768 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjzmn\" (UniqueName: \"kubernetes.io/projected/60146d46-3d75-41de-8d5a-3dba67f81a88-kube-api-access-gjzmn\") pod \"metrics-server-d7c4f5f87-jx56r\" (UID: \"60146d46-3d75-41de-8d5a-3dba67f81a88\") " pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.714649 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.714634 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:21.839360 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:21.839330 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d7c4f5f87-jx56r"] Jul 10 16:41:21.843571 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:41:21.843544 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60146d46_3d75_41de_8d5a_3dba67f81a88.slice/crio-241327c0774c5db3913a5f3d24f8591c6666cc4f76e95ab936f4a09082f54533 WatchSource:0}: Error finding container 241327c0774c5db3913a5f3d24f8591c6666cc4f76e95ab936f4a09082f54533: Status 404 returned error can't find the container with id 241327c0774c5db3913a5f3d24f8591c6666cc4f76e95ab936f4a09082f54533 Jul 10 16:41:22.127560 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:22.127531 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77d699fcbb-vqkjx" Jul 10 16:41:22.215487 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:22.215456 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" event={"ID":"60146d46-3d75-41de-8d5a-3dba67f81a88","Type":"ContainerStarted","Data":"241327c0774c5db3913a5f3d24f8591c6666cc4f76e95ab936f4a09082f54533"} Jul 10 16:41:23.648605 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:23.648581 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:23.648934 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:23.648647 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:23.653583 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:23.653564 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:24.224705 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:24.224669 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" event={"ID":"60146d46-3d75-41de-8d5a-3dba67f81a88","Type":"ContainerStarted","Data":"1eaf3dd9786f6c0a950f23565927e217075cc84b7bb1c0461c482451c961fcca"} Jul 10 16:41:24.228669 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:24.228640 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:41:24.243639 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:24.243599 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" podStartSLOduration=1.507355425 podStartE2EDuration="3.243584702s" totalImagesPullingTime="1.736229277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:41:24.243371587 +0000 UTC m=+303.665248078" watchObservedRunningTime="2026-07-10 16:41:24.243584702 +0000 UTC m=+303.665461168" Jul 10 16:41:24.861822 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:24.861770 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" podUID="2411ecc8-9b66-4410-a942-74fd98224442" containerName="registry" containerID="cri-o://2182956b61eeb7640cf460adec8ba44e8a16149caba3ffff5ee176b09a08ae23" gracePeriod=30 Jul 10 16:41:25.109598 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.109571 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:41:25.134022 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.133948 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2411ecc8-9b66-4410-a942-74fd98224442-ca-trust-extracted\") pod \"2411ecc8-9b66-4410-a942-74fd98224442\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " Jul 10 16:41:25.134022 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.134006 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-bound-sa-token\") pod \"2411ecc8-9b66-4410-a942-74fd98224442\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " Jul 10 16:41:25.134186 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.134058 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2411ecc8-9b66-4410-a942-74fd98224442-registry-certificates\") pod \"2411ecc8-9b66-4410-a942-74fd98224442\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " Jul 10 16:41:25.134186 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.134084 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2411ecc8-9b66-4410-a942-74fd98224442-installation-pull-secrets\") pod \"2411ecc8-9b66-4410-a942-74fd98224442\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " Jul 10 16:41:25.134186 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.134120 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2411ecc8-9b66-4410-a942-74fd98224442-trusted-ca\") pod \"2411ecc8-9b66-4410-a942-74fd98224442\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " Jul 10 16:41:25.134186 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.134148 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqzsr\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-kube-api-access-sqzsr\") pod \"2411ecc8-9b66-4410-a942-74fd98224442\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " Jul 10 16:41:25.134369 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.134199 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls\") pod \"2411ecc8-9b66-4410-a942-74fd98224442\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " Jul 10 16:41:25.134369 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.134241 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2411ecc8-9b66-4410-a942-74fd98224442-image-registry-private-configuration\") pod \"2411ecc8-9b66-4410-a942-74fd98224442\" (UID: \"2411ecc8-9b66-4410-a942-74fd98224442\") " Jul 10 16:41:25.134868 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.134837 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2411ecc8-9b66-4410-a942-74fd98224442-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2411ecc8-9b66-4410-a942-74fd98224442" (UID: "2411ecc8-9b66-4410-a942-74fd98224442"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:41:25.136921 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.136896 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2411ecc8-9b66-4410-a942-74fd98224442" (UID: "2411ecc8-9b66-4410-a942-74fd98224442"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:41:25.137147 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.137110 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2411ecc8-9b66-4410-a942-74fd98224442-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2411ecc8-9b66-4410-a942-74fd98224442" (UID: "2411ecc8-9b66-4410-a942-74fd98224442"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:41:25.137297 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.137142 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2411ecc8-9b66-4410-a942-74fd98224442-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2411ecc8-9b66-4410-a942-74fd98224442" (UID: "2411ecc8-9b66-4410-a942-74fd98224442"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:41:25.137297 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.137162 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-kube-api-access-sqzsr" (OuterVolumeSpecName: "kube-api-access-sqzsr") pod "2411ecc8-9b66-4410-a942-74fd98224442" (UID: "2411ecc8-9b66-4410-a942-74fd98224442"). InnerVolumeSpecName "kube-api-access-sqzsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:41:25.137758 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.137721 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2411ecc8-9b66-4410-a942-74fd98224442-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2411ecc8-9b66-4410-a942-74fd98224442" (UID: "2411ecc8-9b66-4410-a942-74fd98224442"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:41:25.138804 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.138777 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2411ecc8-9b66-4410-a942-74fd98224442" (UID: "2411ecc8-9b66-4410-a942-74fd98224442"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:41:25.143566 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.143544 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2411ecc8-9b66-4410-a942-74fd98224442-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2411ecc8-9b66-4410-a942-74fd98224442" (UID: "2411ecc8-9b66-4410-a942-74fd98224442"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:41:25.229879 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.229843 2585 generic.go:356] "Generic (PLEG): container finished" podID="2411ecc8-9b66-4410-a942-74fd98224442" containerID="2182956b61eeb7640cf460adec8ba44e8a16149caba3ffff5ee176b09a08ae23" exitCode=0 Jul 10 16:41:25.230072 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.229879 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" event={"ID":"2411ecc8-9b66-4410-a942-74fd98224442","Type":"ContainerDied","Data":"2182956b61eeb7640cf460adec8ba44e8a16149caba3ffff5ee176b09a08ae23"} Jul 10 16:41:25.230072 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.229894 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" Jul 10 16:41:25.230072 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.229982 2585 scope.go:117] "RemoveContainer" containerID="2182956b61eeb7640cf460adec8ba44e8a16149caba3ffff5ee176b09a08ae23" Jul 10 16:41:25.230214 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.229968 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d74bffcbd-cvthk" event={"ID":"2411ecc8-9b66-4410-a942-74fd98224442","Type":"ContainerDied","Data":"e0d4e91b0ad48329cfd05819de2b0596e5882ffcac580cadaed6ee5209d07df9"} Jul 10 16:41:25.234910 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.234884 2585 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2411ecc8-9b66-4410-a942-74fd98224442-registry-certificates\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:41:25.235025 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.234912 2585 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2411ecc8-9b66-4410-a942-74fd98224442-installation-pull-secrets\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:41:25.235025 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.234928 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2411ecc8-9b66-4410-a942-74fd98224442-trusted-ca\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:41:25.235025 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.234942 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqzsr\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-kube-api-access-sqzsr\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:41:25.235025 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.234957 2585 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-registry-tls\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:41:25.235025 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.234971 2585 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2411ecc8-9b66-4410-a942-74fd98224442-image-registry-private-configuration\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:41:25.235025 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.234986 2585 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2411ecc8-9b66-4410-a942-74fd98224442-ca-trust-extracted\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:41:25.235025 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.235000 2585 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2411ecc8-9b66-4410-a942-74fd98224442-bound-sa-token\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:41:25.238862 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.238845 2585 scope.go:117] "RemoveContainer" containerID="2182956b61eeb7640cf460adec8ba44e8a16149caba3ffff5ee176b09a08ae23" Jul 10 16:41:25.239131 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:41:25.239113 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2182956b61eeb7640cf460adec8ba44e8a16149caba3ffff5ee176b09a08ae23\": container with ID starting with 2182956b61eeb7640cf460adec8ba44e8a16149caba3ffff5ee176b09a08ae23 not found: ID does not exist" containerID="2182956b61eeb7640cf460adec8ba44e8a16149caba3ffff5ee176b09a08ae23" Jul 10 16:41:25.239180 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.239140 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2182956b61eeb7640cf460adec8ba44e8a16149caba3ffff5ee176b09a08ae23"} err="failed to get container status \"2182956b61eeb7640cf460adec8ba44e8a16149caba3ffff5ee176b09a08ae23\": rpc error: code = NotFound desc = could not find container \"2182956b61eeb7640cf460adec8ba44e8a16149caba3ffff5ee176b09a08ae23\": container with ID starting with 2182956b61eeb7640cf460adec8ba44e8a16149caba3ffff5ee176b09a08ae23 not found: ID does not exist" Jul 10 16:41:25.254948 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.254926 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d74bffcbd-cvthk"] Jul 10 16:41:25.258947 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.258928 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7d74bffcbd-cvthk"] Jul 10 16:41:25.324628 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:25.324596 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2411ecc8-9b66-4410-a942-74fd98224442" path="/var/lib/kubelet/pods/2411ecc8-9b66-4410-a942-74fd98224442/volumes" Jul 10 16:41:27.901904 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:27.901872 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57fdf74bf-6878s"] Jul 10 16:41:27.902260 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:27.902124 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2411ecc8-9b66-4410-a942-74fd98224442" containerName="registry" Jul 10 16:41:27.902260 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:27.902135 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2411ecc8-9b66-4410-a942-74fd98224442" containerName="registry" Jul 10 16:41:27.902260 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:27.902191 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2411ecc8-9b66-4410-a942-74fd98224442" containerName="registry" Jul 10 16:41:27.907057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:27.907036 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:27.915308 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:27.915282 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57fdf74bf-6878s"] Jul 10 16:41:27.956063 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:27.956042 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-trusted-ca-bundle\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:27.956179 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:27.956075 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-oauth-serving-cert\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:27.956179 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:27.956105 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-console-config\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:27.956179 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:27.956128 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bad56be2-229a-4c3e-81ba-3f629927ae16-console-oauth-config\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:27.956179 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:27.956172 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdxbh\" (UniqueName: \"kubernetes.io/projected/bad56be2-229a-4c3e-81ba-3f629927ae16-kube-api-access-tdxbh\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:27.956338 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:27.956210 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bad56be2-229a-4c3e-81ba-3f629927ae16-console-serving-cert\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:27.956338 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:27.956242 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-service-ca\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:28.056486 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:28.056461 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-console-config\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:28.056598 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:28.056491 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bad56be2-229a-4c3e-81ba-3f629927ae16-console-oauth-config\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:28.056598 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:28.056507 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdxbh\" (UniqueName: \"kubernetes.io/projected/bad56be2-229a-4c3e-81ba-3f629927ae16-kube-api-access-tdxbh\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:28.056598 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:28.056540 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bad56be2-229a-4c3e-81ba-3f629927ae16-console-serving-cert\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:28.056598 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:28.056572 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-service-ca\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:28.056831 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:28.056603 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-trusted-ca-bundle\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:28.056831 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:28.056625 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-oauth-serving-cert\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:28.057323 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:28.057271 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-console-config\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:28.057436 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:28.057326 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-service-ca\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:28.057436 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:28.057329 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-oauth-serving-cert\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:28.057547 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:28.057503 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-trusted-ca-bundle\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:28.058975 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:28.058956 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bad56be2-229a-4c3e-81ba-3f629927ae16-console-oauth-config\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:28.059083 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:28.059064 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bad56be2-229a-4c3e-81ba-3f629927ae16-console-serving-cert\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:28.065383 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:28.065356 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdxbh\" (UniqueName: \"kubernetes.io/projected/bad56be2-229a-4c3e-81ba-3f629927ae16-kube-api-access-tdxbh\") pod \"console-57fdf74bf-6878s\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:28.215811 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:28.215729 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:28.341319 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:28.341290 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57fdf74bf-6878s"] Jul 10 16:41:28.344240 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:41:28.344214 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad56be2_229a_4c3e_81ba_3f629927ae16.slice/crio-407d6eb10fcfd433f1c077135870e1dd4e07ac43b473a7786cc9837777dbe660 WatchSource:0}: Error finding container 407d6eb10fcfd433f1c077135870e1dd4e07ac43b473a7786cc9837777dbe660: Status 404 returned error can't find the container with id 407d6eb10fcfd433f1c077135870e1dd4e07ac43b473a7786cc9837777dbe660 Jul 10 16:41:29.246807 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:29.246769 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57fdf74bf-6878s" event={"ID":"bad56be2-229a-4c3e-81ba-3f629927ae16","Type":"ContainerStarted","Data":"5542fdba1b2536bafd2c44ee6e9e52360491219a9c0f5f63fcd304310fbba2b0"} Jul 10 16:41:29.247241 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:29.246863 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57fdf74bf-6878s" event={"ID":"bad56be2-229a-4c3e-81ba-3f629927ae16","Type":"ContainerStarted","Data":"407d6eb10fcfd433f1c077135870e1dd4e07ac43b473a7786cc9837777dbe660"} Jul 10 16:41:29.265843 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:29.265792 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57fdf74bf-6878s" podStartSLOduration=2.265777838 podStartE2EDuration="2.265777838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:41:29.263961336 +0000 UTC m=+308.685837825" watchObservedRunningTime="2026-07-10 16:41:29.265777838 +0000 UTC m=+308.687654305" Jul 10 16:41:34.320017 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:34.319976 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:41:38.216855 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:38.216812 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:38.217285 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:38.216907 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:38.221822 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:38.221797 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:38.283621 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:38.283590 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:41:38.331283 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:38.331237 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ff59fbb78-m4ccl"] Jul 10 16:41:41.715765 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:41.715715 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:41:41.716198 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:41:41.715810 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:42:01.357803 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:01.357770 2585 generic.go:356] "Generic (PLEG): container finished" podID="f6f755a9-cc53-4a26-a3ea-8054a806bf74" containerID="d47c769d9eca1bd01a4ebd82d81f779d2541691184bf8bbcba8d2331cb472463" exitCode=0 Jul 10 16:42:01.358217 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:01.357826 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf" event={"ID":"f6f755a9-cc53-4a26-a3ea-8054a806bf74","Type":"ContainerDied","Data":"d47c769d9eca1bd01a4ebd82d81f779d2541691184bf8bbcba8d2331cb472463"} Jul 10 16:42:01.358217 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:01.358160 2585 scope.go:117] "RemoveContainer" containerID="d47c769d9eca1bd01a4ebd82d81f779d2541691184bf8bbcba8d2331cb472463" Jul 10 16:42:01.720904 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:01.720875 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:42:01.724572 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:01.724543 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-d7c4f5f87-jx56r" Jul 10 16:42:02.363838 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:02.363797 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jf7sf" event={"ID":"f6f755a9-cc53-4a26-a3ea-8054a806bf74","Type":"ContainerStarted","Data":"7192269c3c4104ac8082ad143ca81be4fd62b80c538ee5f212cda851fa506815"} Jul 10 16:42:03.364689 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.364646 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6ff59fbb78-m4ccl" podUID="2a42eb87-0ac1-4d9f-a7f7-2822001805e3" containerName="console" containerID="cri-o://dca10cad899bbbe9fff44dbbc68bb29b20d9ab05b5414627dc55c0c905c4638f" gracePeriod=15 Jul 10 16:42:03.608872 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.608847 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:42:03.748402 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.748309 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-trusted-ca-bundle\") pod \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " Jul 10 16:42:03.748402 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.748353 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-oauth-serving-cert\") pod \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " Jul 10 16:42:03.748402 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.748387 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-console-oauth-config\") pod \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " Jul 10 16:42:03.748648 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.748413 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-console-config\") pod \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " Jul 10 16:42:03.748648 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.748430 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-console-serving-cert\") pod \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " Jul 10 16:42:03.748648 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.748459 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-service-ca\") pod \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " Jul 10 16:42:03.748648 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.748494 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzw4q\" (UniqueName: \"kubernetes.io/projected/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-kube-api-access-xzw4q\") pod \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\" (UID: \"2a42eb87-0ac1-4d9f-a7f7-2822001805e3\") " Jul 10 16:42:03.748861 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.748785 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2a42eb87-0ac1-4d9f-a7f7-2822001805e3" (UID: "2a42eb87-0ac1-4d9f-a7f7-2822001805e3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:42:03.749070 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.749045 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2a42eb87-0ac1-4d9f-a7f7-2822001805e3" (UID: "2a42eb87-0ac1-4d9f-a7f7-2822001805e3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:42:03.749137 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.749054 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-console-config" (OuterVolumeSpecName: "console-config") pod "2a42eb87-0ac1-4d9f-a7f7-2822001805e3" (UID: "2a42eb87-0ac1-4d9f-a7f7-2822001805e3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:42:03.749186 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.749126 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-service-ca" (OuterVolumeSpecName: "service-ca") pod "2a42eb87-0ac1-4d9f-a7f7-2822001805e3" (UID: "2a42eb87-0ac1-4d9f-a7f7-2822001805e3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:42:03.750900 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.750878 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2a42eb87-0ac1-4d9f-a7f7-2822001805e3" (UID: "2a42eb87-0ac1-4d9f-a7f7-2822001805e3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:42:03.751195 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.751175 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2a42eb87-0ac1-4d9f-a7f7-2822001805e3" (UID: "2a42eb87-0ac1-4d9f-a7f7-2822001805e3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:42:03.751248 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.751192 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-kube-api-access-xzw4q" (OuterVolumeSpecName: "kube-api-access-xzw4q") pod "2a42eb87-0ac1-4d9f-a7f7-2822001805e3" (UID: "2a42eb87-0ac1-4d9f-a7f7-2822001805e3"). InnerVolumeSpecName "kube-api-access-xzw4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:42:03.849119 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.849089 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-trusted-ca-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:42:03.849119 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.849118 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-oauth-serving-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:42:03.849119 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.849127 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-console-oauth-config\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:42:03.849340 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.849137 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-console-config\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:42:03.849340 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.849145 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-console-serving-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:42:03.849340 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.849154 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-service-ca\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:42:03.849340 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:03.849161 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzw4q\" (UniqueName: \"kubernetes.io/projected/2a42eb87-0ac1-4d9f-a7f7-2822001805e3-kube-api-access-xzw4q\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:42:04.371700 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:04.371665 2585 generic.go:356] "Generic (PLEG): container finished" podID="2a42eb87-0ac1-4d9f-a7f7-2822001805e3" containerID="dca10cad899bbbe9fff44dbbc68bb29b20d9ab05b5414627dc55c0c905c4638f" exitCode=0 Jul 10 16:42:04.372149 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:04.371711 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff59fbb78-m4ccl" Jul 10 16:42:04.372149 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:04.371737 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff59fbb78-m4ccl" event={"ID":"2a42eb87-0ac1-4d9f-a7f7-2822001805e3","Type":"ContainerDied","Data":"dca10cad899bbbe9fff44dbbc68bb29b20d9ab05b5414627dc55c0c905c4638f"} Jul 10 16:42:04.372149 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:04.371800 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff59fbb78-m4ccl" event={"ID":"2a42eb87-0ac1-4d9f-a7f7-2822001805e3","Type":"ContainerDied","Data":"a2ca38129cd1c9b69d7615ec799b4a644394a38891698f66e5b1310bfdfa6ac8"} Jul 10 16:42:04.372149 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:04.371820 2585 scope.go:117] "RemoveContainer" containerID="dca10cad899bbbe9fff44dbbc68bb29b20d9ab05b5414627dc55c0c905c4638f" Jul 10 16:42:04.380839 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:04.380816 2585 scope.go:117] "RemoveContainer" containerID="dca10cad899bbbe9fff44dbbc68bb29b20d9ab05b5414627dc55c0c905c4638f" Jul 10 16:42:04.381138 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:42:04.381121 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca10cad899bbbe9fff44dbbc68bb29b20d9ab05b5414627dc55c0c905c4638f\": container with ID starting with dca10cad899bbbe9fff44dbbc68bb29b20d9ab05b5414627dc55c0c905c4638f not found: ID does not exist" containerID="dca10cad899bbbe9fff44dbbc68bb29b20d9ab05b5414627dc55c0c905c4638f" Jul 10 16:42:04.381181 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:04.381146 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca10cad899bbbe9fff44dbbc68bb29b20d9ab05b5414627dc55c0c905c4638f"} err="failed to get container status \"dca10cad899bbbe9fff44dbbc68bb29b20d9ab05b5414627dc55c0c905c4638f\": rpc error: code = NotFound desc = could not find container \"dca10cad899bbbe9fff44dbbc68bb29b20d9ab05b5414627dc55c0c905c4638f\": container with ID starting with dca10cad899bbbe9fff44dbbc68bb29b20d9ab05b5414627dc55c0c905c4638f not found: ID does not exist" Jul 10 16:42:04.393794 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:04.393739 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ff59fbb78-m4ccl"] Jul 10 16:42:04.397029 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:04.396999 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6ff59fbb78-m4ccl"] Jul 10 16:42:05.324258 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:05.324223 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a42eb87-0ac1-4d9f-a7f7-2822001805e3" path="/var/lib/kubelet/pods/2a42eb87-0ac1-4d9f-a7f7-2822001805e3/volumes" Jul 10 16:42:34.919138 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:34.919079 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs\") pod \"network-metrics-daemon-6c49f\" (UID: \"08a4d929-2c36-40ac-a3e6-8c5caf0c1788\") " pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:42:34.921559 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:34.921531 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08a4d929-2c36-40ac-a3e6-8c5caf0c1788-metrics-certs\") pod \"network-metrics-daemon-6c49f\" (UID: \"08a4d929-2c36-40ac-a3e6-8c5caf0c1788\") " pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:42:35.224369 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:35.224283 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bc5mm\"" Jul 10 16:42:35.232153 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:35.232129 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6c49f" Jul 10 16:42:35.361835 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:35.361800 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6c49f"] Jul 10 16:42:35.365076 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:42:35.365046 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a4d929_2c36_40ac_a3e6_8c5caf0c1788.slice/crio-b87ce39689d0896e5ac75e806428ae3d46c2d8ed2fe5b90e898825bd5fc24b92 WatchSource:0}: Error finding container b87ce39689d0896e5ac75e806428ae3d46c2d8ed2fe5b90e898825bd5fc24b92: Status 404 returned error can't find the container with id b87ce39689d0896e5ac75e806428ae3d46c2d8ed2fe5b90e898825bd5fc24b92 Jul 10 16:42:35.367009 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:35.366992 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 16:42:35.485890 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:35.485799 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6c49f" event={"ID":"08a4d929-2c36-40ac-a3e6-8c5caf0c1788","Type":"ContainerStarted","Data":"b87ce39689d0896e5ac75e806428ae3d46c2d8ed2fe5b90e898825bd5fc24b92"} Jul 10 16:42:37.495146 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:37.495106 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6c49f" event={"ID":"08a4d929-2c36-40ac-a3e6-8c5caf0c1788","Type":"ContainerStarted","Data":"7e47c47d8a7fc5cdfde2757c00382193c64dc6641e6aa866cd86baa50e5f9662"} Jul 10 16:42:37.495583 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:37.495159 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6c49f" event={"ID":"08a4d929-2c36-40ac-a3e6-8c5caf0c1788","Type":"ContainerStarted","Data":"35e77b9d8a2e94b0e8c8ed281ec95ed2704e9191897b153ab6dfef7816910d42"} Jul 10 16:42:37.513876 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:37.513831 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6c49f" podStartSLOduration=375.429304346 podStartE2EDuration="6m16.513814967s" totalImagesPullingTime="1.084510621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:42:37.511639196 +0000 UTC m=+376.933515697" watchObservedRunningTime="2026-07-10 16:42:37.513814967 +0000 UTC m=+376.935691435" Jul 10 16:42:39.242866 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.242819 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 16:42:39.243263 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.243180 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a42eb87-0ac1-4d9f-a7f7-2822001805e3" containerName="console" Jul 10 16:42:39.243263 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.243195 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a42eb87-0ac1-4d9f-a7f7-2822001805e3" containerName="console" Jul 10 16:42:39.243263 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.243252 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a42eb87-0ac1-4d9f-a7f7-2822001805e3" containerName="console" Jul 10 16:42:39.246715 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.246699 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.249631 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.249593 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 10 16:42:39.249631 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.249626 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 10 16:42:39.249870 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.249649 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 10 16:42:39.249870 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.249610 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 10 16:42:39.249870 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.249611 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 10 16:42:39.250095 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.250078 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 10 16:42:39.250095 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.250089 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 10 16:42:39.250211 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.250103 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 10 16:42:39.251391 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.251370 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-86x2l\"" Jul 10 16:42:39.255098 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.255078 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 10 16:42:39.261487 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.261454 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 16:42:39.354050 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.354003 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.354050 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.354051 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.354317 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.354113 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72e7ded4-2f33-4ac5-918b-04a2caee91e1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.354317 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.354145 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.354317 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.354170 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72e7ded4-2f33-4ac5-918b-04a2caee91e1-config-out\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.354317 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.354196 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-web-config\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.354317 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.354252 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.354317 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.354280 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/72e7ded4-2f33-4ac5-918b-04a2caee91e1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.354544 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.354320 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48pdk\" (UniqueName: \"kubernetes.io/projected/72e7ded4-2f33-4ac5-918b-04a2caee91e1-kube-api-access-48pdk\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.354544 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.354356 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72e7ded4-2f33-4ac5-918b-04a2caee91e1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.354544 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.354379 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e7ded4-2f33-4ac5-918b-04a2caee91e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.354544 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.354412 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-config-volume\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.354544 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.354449 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.455379 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.455330 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.455557 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.455389 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/72e7ded4-2f33-4ac5-918b-04a2caee91e1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.455557 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.455437 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48pdk\" (UniqueName: \"kubernetes.io/projected/72e7ded4-2f33-4ac5-918b-04a2caee91e1-kube-api-access-48pdk\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.455681 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.455603 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72e7ded4-2f33-4ac5-918b-04a2caee91e1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.455681 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.455641 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e7ded4-2f33-4ac5-918b-04a2caee91e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.455831 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.455679 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-config-volume\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.455831 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.455705 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.455931 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.455851 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.456316 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.456286 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.456542 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.456521 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72e7ded4-2f33-4ac5-918b-04a2caee91e1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.456710 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.456688 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.456869 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.456851 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72e7ded4-2f33-4ac5-918b-04a2caee91e1-config-out\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.456999 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.456984 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-web-config\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.458387 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.458358 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.458939 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.458914 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.459169 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.459150 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/72e7ded4-2f33-4ac5-918b-04a2caee91e1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.459763 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.459722 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.459968 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.459901 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72e7ded4-2f33-4ac5-918b-04a2caee91e1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.460233 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.457001 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e7ded4-2f33-4ac5-918b-04a2caee91e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.460233 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.460191 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72e7ded4-2f33-4ac5-918b-04a2caee91e1-config-out\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.460609 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.460584 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72e7ded4-2f33-4ac5-918b-04a2caee91e1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.461824 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.461106 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.465270 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.463070 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.465270 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.463421 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-web-config\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.465270 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.465027 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48pdk\" (UniqueName: \"kubernetes.io/projected/72e7ded4-2f33-4ac5-918b-04a2caee91e1-kube-api-access-48pdk\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.465954 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.465931 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/72e7ded4-2f33-4ac5-918b-04a2caee91e1-config-volume\") pod \"alertmanager-main-0\" (UID: \"72e7ded4-2f33-4ac5-918b-04a2caee91e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.556986 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.556904 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 16:42:39.693922 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:39.693887 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 16:42:39.696864 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:42:39.696836 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e7ded4_2f33_4ac5_918b_04a2caee91e1.slice/crio-8484bdd6a651632ed6c86de916b7c358b6f13831cb452e920e29b5fcb8192646 WatchSource:0}: Error finding container 8484bdd6a651632ed6c86de916b7c358b6f13831cb452e920e29b5fcb8192646: Status 404 returned error can't find the container with id 8484bdd6a651632ed6c86de916b7c358b6f13831cb452e920e29b5fcb8192646 Jul 10 16:42:40.508375 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:40.508334 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72e7ded4-2f33-4ac5-918b-04a2caee91e1","Type":"ContainerStarted","Data":"8484bdd6a651632ed6c86de916b7c358b6f13831cb452e920e29b5fcb8192646"} Jul 10 16:42:41.513824 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:41.513783 2585 generic.go:356] "Generic (PLEG): container finished" podID="72e7ded4-2f33-4ac5-918b-04a2caee91e1" containerID="10c889cd7c383f1e93dab06024c39f94c97eb92ad37d57b3176fc312c6d01d73" exitCode=0 Jul 10 16:42:41.514181 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:41.513830 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72e7ded4-2f33-4ac5-918b-04a2caee91e1","Type":"ContainerDied","Data":"10c889cd7c383f1e93dab06024c39f94c97eb92ad37d57b3176fc312c6d01d73"} Jul 10 16:42:44.528944 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:44.528906 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72e7ded4-2f33-4ac5-918b-04a2caee91e1","Type":"ContainerStarted","Data":"6d13f3aeb49096c11b7182b8a30ea3b05622de1109928106100a2cbd5b60d5ad"} Jul 10 16:42:44.529323 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:44.528977 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72e7ded4-2f33-4ac5-918b-04a2caee91e1","Type":"ContainerStarted","Data":"e57a140b3e9efc004fe9bd16c964b4faf0cb97b5ddea8889b47f045c1afcb357"} Jul 10 16:42:44.529323 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:44.529003 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72e7ded4-2f33-4ac5-918b-04a2caee91e1","Type":"ContainerStarted","Data":"111a844fbfd4f8781be37499fcbb5dbe713e978a3022c0d9ad946c01101ede65"} Jul 10 16:42:44.529323 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:44.529028 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72e7ded4-2f33-4ac5-918b-04a2caee91e1","Type":"ContainerStarted","Data":"b4dd87bd74c08de099fc0e0efbf672417e6c0a79eb950ffd4fa53364fa540ef6"} Jul 10 16:42:44.529323 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:44.529051 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72e7ded4-2f33-4ac5-918b-04a2caee91e1","Type":"ContainerStarted","Data":"550fb00e26aafd610440cacd346b7b95a9980c1fd08d5cd158ca0906067b3c01"} Jul 10 16:42:45.535483 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:45.535443 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72e7ded4-2f33-4ac5-918b-04a2caee91e1","Type":"ContainerStarted","Data":"cf8a4f936d091044f7c9dff18ece147cbdbc47540316d78ba469135ad00e145f"} Jul 10 16:42:45.563279 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:42:45.563062 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.250194132 podStartE2EDuration="6.563041384s" totalImagesPullingTime="4.293358704s" totalInitContainerRuntime="19.488548ms" isStatefulPod=false podCreationTimestamp="2026-07-10 16:42:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:42:45.561182523 +0000 UTC m=+384.983058991" watchObservedRunningTime="2026-07-10 16:42:45.563041384 +0000 UTC m=+384.984917852" Jul 10 16:43:05.374489 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:05.374449 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57fdf74bf-6878s"] Jul 10 16:43:30.396356 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.396316 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57fdf74bf-6878s" podUID="bad56be2-229a-4c3e-81ba-3f629927ae16" containerName="console" containerID="cri-o://5542fdba1b2536bafd2c44ee6e9e52360491219a9c0f5f63fcd304310fbba2b0" gracePeriod=15 Jul 10 16:43:30.635260 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.635229 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:43:30.711234 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.711154 2585 generic.go:356] "Generic (PLEG): container finished" podID="bad56be2-229a-4c3e-81ba-3f629927ae16" containerID="5542fdba1b2536bafd2c44ee6e9e52360491219a9c0f5f63fcd304310fbba2b0" exitCode=0 Jul 10 16:43:30.711234 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.711196 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57fdf74bf-6878s" Jul 10 16:43:30.711401 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.711244 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57fdf74bf-6878s" event={"ID":"bad56be2-229a-4c3e-81ba-3f629927ae16","Type":"ContainerDied","Data":"5542fdba1b2536bafd2c44ee6e9e52360491219a9c0f5f63fcd304310fbba2b0"} Jul 10 16:43:30.711401 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.711295 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57fdf74bf-6878s" event={"ID":"bad56be2-229a-4c3e-81ba-3f629927ae16","Type":"ContainerDied","Data":"407d6eb10fcfd433f1c077135870e1dd4e07ac43b473a7786cc9837777dbe660"} Jul 10 16:43:30.711401 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.711322 2585 scope.go:117] "RemoveContainer" containerID="5542fdba1b2536bafd2c44ee6e9e52360491219a9c0f5f63fcd304310fbba2b0" Jul 10 16:43:30.720588 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.720570 2585 scope.go:117] "RemoveContainer" containerID="5542fdba1b2536bafd2c44ee6e9e52360491219a9c0f5f63fcd304310fbba2b0" Jul 10 16:43:30.720864 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:43:30.720844 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5542fdba1b2536bafd2c44ee6e9e52360491219a9c0f5f63fcd304310fbba2b0\": container with ID starting with 5542fdba1b2536bafd2c44ee6e9e52360491219a9c0f5f63fcd304310fbba2b0 not found: ID does not exist" containerID="5542fdba1b2536bafd2c44ee6e9e52360491219a9c0f5f63fcd304310fbba2b0" Jul 10 16:43:30.720931 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.720874 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5542fdba1b2536bafd2c44ee6e9e52360491219a9c0f5f63fcd304310fbba2b0"} err="failed to get container status \"5542fdba1b2536bafd2c44ee6e9e52360491219a9c0f5f63fcd304310fbba2b0\": rpc error: code = NotFound desc = could not find container \"5542fdba1b2536bafd2c44ee6e9e52360491219a9c0f5f63fcd304310fbba2b0\": container with ID starting with 5542fdba1b2536bafd2c44ee6e9e52360491219a9c0f5f63fcd304310fbba2b0 not found: ID does not exist" Jul 10 16:43:30.781233 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.781207 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-oauth-serving-cert\") pod \"bad56be2-229a-4c3e-81ba-3f629927ae16\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " Jul 10 16:43:30.781324 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.781251 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bad56be2-229a-4c3e-81ba-3f629927ae16-console-oauth-config\") pod \"bad56be2-229a-4c3e-81ba-3f629927ae16\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " Jul 10 16:43:30.781324 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.781289 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-trusted-ca-bundle\") pod \"bad56be2-229a-4c3e-81ba-3f629927ae16\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " Jul 10 16:43:30.781324 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.781317 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-console-config\") pod \"bad56be2-229a-4c3e-81ba-3f629927ae16\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " Jul 10 16:43:30.781445 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.781344 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdxbh\" (UniqueName: \"kubernetes.io/projected/bad56be2-229a-4c3e-81ba-3f629927ae16-kube-api-access-tdxbh\") pod \"bad56be2-229a-4c3e-81ba-3f629927ae16\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " Jul 10 16:43:30.781792 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.781502 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bad56be2-229a-4c3e-81ba-3f629927ae16-console-serving-cert\") pod \"bad56be2-229a-4c3e-81ba-3f629927ae16\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " Jul 10 16:43:30.781792 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.781552 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-service-ca\") pod \"bad56be2-229a-4c3e-81ba-3f629927ae16\" (UID: \"bad56be2-229a-4c3e-81ba-3f629927ae16\") " Jul 10 16:43:30.781792 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.781634 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bad56be2-229a-4c3e-81ba-3f629927ae16" (UID: "bad56be2-229a-4c3e-81ba-3f629927ae16"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:43:30.781792 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.781739 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-console-config" (OuterVolumeSpecName: "console-config") pod "bad56be2-229a-4c3e-81ba-3f629927ae16" (UID: "bad56be2-229a-4c3e-81ba-3f629927ae16"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:43:30.781999 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.781832 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bad56be2-229a-4c3e-81ba-3f629927ae16" (UID: "bad56be2-229a-4c3e-81ba-3f629927ae16"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:43:30.781999 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.781847 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-console-config\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:43:30.781999 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.781867 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-oauth-serving-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:43:30.781999 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.781959 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-service-ca" (OuterVolumeSpecName: "service-ca") pod "bad56be2-229a-4c3e-81ba-3f629927ae16" (UID: "bad56be2-229a-4c3e-81ba-3f629927ae16"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:43:30.783462 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.783429 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bad56be2-229a-4c3e-81ba-3f629927ae16-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bad56be2-229a-4c3e-81ba-3f629927ae16" (UID: "bad56be2-229a-4c3e-81ba-3f629927ae16"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:43:30.783552 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.783486 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad56be2-229a-4c3e-81ba-3f629927ae16-kube-api-access-tdxbh" (OuterVolumeSpecName: "kube-api-access-tdxbh") pod "bad56be2-229a-4c3e-81ba-3f629927ae16" (UID: "bad56be2-229a-4c3e-81ba-3f629927ae16"). InnerVolumeSpecName "kube-api-access-tdxbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:43:30.783552 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.783495 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bad56be2-229a-4c3e-81ba-3f629927ae16-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bad56be2-229a-4c3e-81ba-3f629927ae16" (UID: "bad56be2-229a-4c3e-81ba-3f629927ae16"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:43:30.882377 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.882334 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bad56be2-229a-4c3e-81ba-3f629927ae16-console-oauth-config\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:43:30.882377 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.882371 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-trusted-ca-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:43:30.882377 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.882385 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdxbh\" (UniqueName: \"kubernetes.io/projected/bad56be2-229a-4c3e-81ba-3f629927ae16-kube-api-access-tdxbh\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:43:30.882610 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.882398 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bad56be2-229a-4c3e-81ba-3f629927ae16-console-serving-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:43:30.882610 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:30.882411 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bad56be2-229a-4c3e-81ba-3f629927ae16-service-ca\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:43:31.032950 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:31.032922 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57fdf74bf-6878s"] Jul 10 16:43:31.038932 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:31.038906 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57fdf74bf-6878s"] Jul 10 16:43:31.324717 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:43:31.324624 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bad56be2-229a-4c3e-81ba-3f629927ae16" path="/var/lib/kubelet/pods/bad56be2-229a-4c3e-81ba-3f629927ae16/volumes" Jul 10 16:44:04.232215 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.232178 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4"] Jul 10 16:44:04.232632 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.232482 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bad56be2-229a-4c3e-81ba-3f629927ae16" containerName="console" Jul 10 16:44:04.232632 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.232494 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bad56be2-229a-4c3e-81ba-3f629927ae16" containerName="console" Jul 10 16:44:04.232632 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.232545 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="bad56be2-229a-4c3e-81ba-3f629927ae16" containerName="console" Jul 10 16:44:04.235652 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.235634 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4" Jul 10 16:44:04.239100 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.239083 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 16:44:04.239470 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.239455 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-25b9w\"" Jul 10 16:44:04.240463 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.240445 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 16:44:04.249771 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.249724 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4"] Jul 10 16:44:04.252652 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.252629 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbc09fb8-673a-4502-bfe2-8fd0b2522b5a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4\" (UID: \"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4" Jul 10 16:44:04.252785 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.252695 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbc09fb8-673a-4502-bfe2-8fd0b2522b5a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4\" (UID: \"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4" Jul 10 16:44:04.252785 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.252731 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hngk5\" (UniqueName: \"kubernetes.io/projected/bbc09fb8-673a-4502-bfe2-8fd0b2522b5a-kube-api-access-hngk5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4\" (UID: \"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4" Jul 10 16:44:04.353897 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.353862 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbc09fb8-673a-4502-bfe2-8fd0b2522b5a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4\" (UID: \"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4" Jul 10 16:44:04.354056 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.353916 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbc09fb8-673a-4502-bfe2-8fd0b2522b5a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4\" (UID: \"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4" Jul 10 16:44:04.354056 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.353940 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hngk5\" (UniqueName: \"kubernetes.io/projected/bbc09fb8-673a-4502-bfe2-8fd0b2522b5a-kube-api-access-hngk5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4\" (UID: \"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4" Jul 10 16:44:04.354282 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.354262 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbc09fb8-673a-4502-bfe2-8fd0b2522b5a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4\" (UID: \"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4" Jul 10 16:44:04.354318 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.354265 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbc09fb8-673a-4502-bfe2-8fd0b2522b5a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4\" (UID: \"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4" Jul 10 16:44:04.362176 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.362144 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hngk5\" (UniqueName: \"kubernetes.io/projected/bbc09fb8-673a-4502-bfe2-8fd0b2522b5a-kube-api-access-hngk5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4\" (UID: \"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4" Jul 10 16:44:04.543857 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.543774 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4" Jul 10 16:44:04.667424 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.667398 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4"] Jul 10 16:44:04.669692 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:44:04.669665 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc09fb8_673a_4502_bfe2_8fd0b2522b5a.slice/crio-0b8830017f2879e3fc4939aaeed98dd376369a012fc59c5987812a5fa7eb0ac1 WatchSource:0}: Error finding container 0b8830017f2879e3fc4939aaeed98dd376369a012fc59c5987812a5fa7eb0ac1: Status 404 returned error can't find the container with id 0b8830017f2879e3fc4939aaeed98dd376369a012fc59c5987812a5fa7eb0ac1 Jul 10 16:44:04.843731 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:04.843642 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4" event={"ID":"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a","Type":"ContainerStarted","Data":"0b8830017f2879e3fc4939aaeed98dd376369a012fc59c5987812a5fa7eb0ac1"} Jul 10 16:44:12.876784 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:12.876721 2585 generic.go:356] "Generic (PLEG): container finished" podID="bbc09fb8-673a-4502-bfe2-8fd0b2522b5a" containerID="b4d8cfe7e5ad82b551bb5e597a8c0ce69f48a105a9972d45cd94a50d248f8a37" exitCode=0 Jul 10 16:44:12.877166 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:12.876813 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4" event={"ID":"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a","Type":"ContainerDied","Data":"b4d8cfe7e5ad82b551bb5e597a8c0ce69f48a105a9972d45cd94a50d248f8a37"} Jul 10 16:44:14.886733 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:14.886703 2585 generic.go:356] "Generic (PLEG): container finished" podID="bbc09fb8-673a-4502-bfe2-8fd0b2522b5a" containerID="2316f15b3c9926ddcc194e5465bb5edaa866c02d27ff06438361224507cb65e4" exitCode=0 Jul 10 16:44:14.887125 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:14.886738 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4" event={"ID":"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a","Type":"ContainerDied","Data":"2316f15b3c9926ddcc194e5465bb5edaa866c02d27ff06438361224507cb65e4"} Jul 10 16:44:24.928828 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:24.928789 2585 generic.go:356] "Generic (PLEG): container finished" podID="bbc09fb8-673a-4502-bfe2-8fd0b2522b5a" containerID="abd3c2b267219532b3be035d022a08f2a1221786ee33f666e71c19ab56e49978" exitCode=0 Jul 10 16:44:24.929229 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:24.928857 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4" event={"ID":"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a","Type":"ContainerDied","Data":"abd3c2b267219532b3be035d022a08f2a1221786ee33f666e71c19ab56e49978"} Jul 10 16:44:26.074059 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:26.074023 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4" Jul 10 16:44:26.138886 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:26.138821 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbc09fb8-673a-4502-bfe2-8fd0b2522b5a-bundle\") pod \"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a\" (UID: \"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a\") " Jul 10 16:44:26.139079 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:26.138924 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hngk5\" (UniqueName: \"kubernetes.io/projected/bbc09fb8-673a-4502-bfe2-8fd0b2522b5a-kube-api-access-hngk5\") pod \"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a\" (UID: \"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a\") " Jul 10 16:44:26.139079 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:26.138953 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbc09fb8-673a-4502-bfe2-8fd0b2522b5a-util\") pod \"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a\" (UID: \"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a\") " Jul 10 16:44:26.139501 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:26.139479 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc09fb8-673a-4502-bfe2-8fd0b2522b5a-bundle" (OuterVolumeSpecName: "bundle") pod "bbc09fb8-673a-4502-bfe2-8fd0b2522b5a" (UID: "bbc09fb8-673a-4502-bfe2-8fd0b2522b5a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:44:26.141324 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:26.141285 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc09fb8-673a-4502-bfe2-8fd0b2522b5a-kube-api-access-hngk5" (OuterVolumeSpecName: "kube-api-access-hngk5") pod "bbc09fb8-673a-4502-bfe2-8fd0b2522b5a" (UID: "bbc09fb8-673a-4502-bfe2-8fd0b2522b5a"). InnerVolumeSpecName "kube-api-access-hngk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:44:26.143739 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:26.143718 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc09fb8-673a-4502-bfe2-8fd0b2522b5a-util" (OuterVolumeSpecName: "util") pod "bbc09fb8-673a-4502-bfe2-8fd0b2522b5a" (UID: "bbc09fb8-673a-4502-bfe2-8fd0b2522b5a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:44:26.240542 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:26.240466 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbc09fb8-673a-4502-bfe2-8fd0b2522b5a-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:44:26.240542 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:26.240493 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hngk5\" (UniqueName: \"kubernetes.io/projected/bbc09fb8-673a-4502-bfe2-8fd0b2522b5a-kube-api-access-hngk5\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:44:26.240542 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:26.240503 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbc09fb8-673a-4502-bfe2-8fd0b2522b5a-util\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:44:26.938209 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:26.938174 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4" event={"ID":"bbc09fb8-673a-4502-bfe2-8fd0b2522b5a","Type":"ContainerDied","Data":"0b8830017f2879e3fc4939aaeed98dd376369a012fc59c5987812a5fa7eb0ac1"} Jul 10 16:44:26.938209 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:26.938197 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404d6k4" Jul 10 16:44:26.938209 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:26.938218 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b8830017f2879e3fc4939aaeed98dd376369a012fc59c5987812a5fa7eb0ac1" Jul 10 16:44:32.216572 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.216532 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6p5qg"] Jul 10 16:44:32.217046 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.216963 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc09fb8-673a-4502-bfe2-8fd0b2522b5a" containerName="extract" Jul 10 16:44:32.217046 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.216983 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc09fb8-673a-4502-bfe2-8fd0b2522b5a" containerName="extract" Jul 10 16:44:32.217046 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.216997 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc09fb8-673a-4502-bfe2-8fd0b2522b5a" containerName="pull" Jul 10 16:44:32.217046 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.217008 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc09fb8-673a-4502-bfe2-8fd0b2522b5a" containerName="pull" Jul 10 16:44:32.217046 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.217022 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc09fb8-673a-4502-bfe2-8fd0b2522b5a" containerName="util" Jul 10 16:44:32.217046 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.217032 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc09fb8-673a-4502-bfe2-8fd0b2522b5a" containerName="util" Jul 10 16:44:32.217225 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.217110 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc09fb8-673a-4502-bfe2-8fd0b2522b5a" containerName="extract" Jul 10 16:44:32.220774 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.220736 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6p5qg" Jul 10 16:44:32.224014 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.223995 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 10 16:44:32.224173 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.224072 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 10 16:44:32.224355 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.224340 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-z7sd5\"" Jul 10 16:44:32.232536 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.232513 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6p5qg"] Jul 10 16:44:32.290331 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.290306 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndlk6\" (UniqueName: \"kubernetes.io/projected/61b16a99-778a-4b42-9cff-5239406660b9-kube-api-access-ndlk6\") pod \"cert-manager-operator-controller-manager-55c444bf86-6p5qg\" (UID: \"61b16a99-778a-4b42-9cff-5239406660b9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6p5qg" Jul 10 16:44:32.290446 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.290339 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61b16a99-778a-4b42-9cff-5239406660b9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6p5qg\" (UID: \"61b16a99-778a-4b42-9cff-5239406660b9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6p5qg" Jul 10 16:44:32.390983 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.390953 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndlk6\" (UniqueName: \"kubernetes.io/projected/61b16a99-778a-4b42-9cff-5239406660b9-kube-api-access-ndlk6\") pod \"cert-manager-operator-controller-manager-55c444bf86-6p5qg\" (UID: \"61b16a99-778a-4b42-9cff-5239406660b9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6p5qg" Jul 10 16:44:32.391121 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.390991 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61b16a99-778a-4b42-9cff-5239406660b9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6p5qg\" (UID: \"61b16a99-778a-4b42-9cff-5239406660b9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6p5qg" Jul 10 16:44:32.391404 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.391384 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61b16a99-778a-4b42-9cff-5239406660b9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6p5qg\" (UID: \"61b16a99-778a-4b42-9cff-5239406660b9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6p5qg" Jul 10 16:44:32.401065 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.401043 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndlk6\" (UniqueName: \"kubernetes.io/projected/61b16a99-778a-4b42-9cff-5239406660b9-kube-api-access-ndlk6\") pod \"cert-manager-operator-controller-manager-55c444bf86-6p5qg\" (UID: \"61b16a99-778a-4b42-9cff-5239406660b9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6p5qg" Jul 10 16:44:32.529806 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.529719 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6p5qg" Jul 10 16:44:32.670142 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.670111 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6p5qg"] Jul 10 16:44:32.673922 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:44:32.673888 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b16a99_778a_4b42_9cff_5239406660b9.slice/crio-bf6b806955557d79866fbd16b5220e558c053d57a142aadf8ba7b8d7e7783db7 WatchSource:0}: Error finding container bf6b806955557d79866fbd16b5220e558c053d57a142aadf8ba7b8d7e7783db7: Status 404 returned error can't find the container with id bf6b806955557d79866fbd16b5220e558c053d57a142aadf8ba7b8d7e7783db7 Jul 10 16:44:32.961967 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:32.961929 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6p5qg" event={"ID":"61b16a99-778a-4b42-9cff-5239406660b9","Type":"ContainerStarted","Data":"bf6b806955557d79866fbd16b5220e558c053d57a142aadf8ba7b8d7e7783db7"} Jul 10 16:44:34.972009 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:34.971965 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6p5qg" event={"ID":"61b16a99-778a-4b42-9cff-5239406660b9","Type":"ContainerStarted","Data":"b05f55d6195a94d0cd5eb1797765cd8014cd341b8d6607e3451d46e7e0c42638"} Jul 10 16:44:34.993352 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:34.993295 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6p5qg" podStartSLOduration=1.5874346990000001 podStartE2EDuration="2.993277583s" totalImagesPullingTime="1.405842884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:44:34.991421864 +0000 UTC m=+494.413298349" watchObservedRunningTime="2026-07-10 16:44:34.993277583 +0000 UTC m=+494.415154052" Jul 10 16:44:35.980819 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:35.980781 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh"] Jul 10 16:44:35.983309 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:35.983288 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh" Jul 10 16:44:35.986213 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:35.986194 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 16:44:35.986985 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:35.986949 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-25b9w\"" Jul 10 16:44:35.987729 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:35.987713 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 16:44:35.993825 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:35.993804 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh"] Jul 10 16:44:36.126608 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.126571 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/914abbd8-ce78-435e-b0ab-84357420340c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh\" (UID: \"914abbd8-ce78-435e-b0ab-84357420340c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh" Jul 10 16:44:36.126796 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.126709 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbts7\" (UniqueName: \"kubernetes.io/projected/914abbd8-ce78-435e-b0ab-84357420340c-kube-api-access-fbts7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh\" (UID: \"914abbd8-ce78-435e-b0ab-84357420340c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh" Jul 10 16:44:36.126796 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.126769 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/914abbd8-ce78-435e-b0ab-84357420340c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh\" (UID: \"914abbd8-ce78-435e-b0ab-84357420340c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh" Jul 10 16:44:36.184116 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.184084 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2hcdv"] Jul 10 16:44:36.186187 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.186169 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-2hcdv" Jul 10 16:44:36.188974 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.188951 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-m6g5g\"" Jul 10 16:44:36.189099 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.188985 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 16:44:36.189262 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.189248 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 16:44:36.196671 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.196646 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2hcdv"] Jul 10 16:44:36.227231 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.227196 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbts7\" (UniqueName: \"kubernetes.io/projected/914abbd8-ce78-435e-b0ab-84357420340c-kube-api-access-fbts7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh\" (UID: \"914abbd8-ce78-435e-b0ab-84357420340c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh" Jul 10 16:44:36.227392 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.227243 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/914abbd8-ce78-435e-b0ab-84357420340c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh\" (UID: \"914abbd8-ce78-435e-b0ab-84357420340c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh" Jul 10 16:44:36.227392 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.227274 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/914abbd8-ce78-435e-b0ab-84357420340c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh\" (UID: \"914abbd8-ce78-435e-b0ab-84357420340c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh" Jul 10 16:44:36.227673 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.227655 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/914abbd8-ce78-435e-b0ab-84357420340c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh\" (UID: \"914abbd8-ce78-435e-b0ab-84357420340c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh" Jul 10 16:44:36.227710 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.227670 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/914abbd8-ce78-435e-b0ab-84357420340c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh\" (UID: \"914abbd8-ce78-435e-b0ab-84357420340c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh" Jul 10 16:44:36.235607 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.235540 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbts7\" (UniqueName: \"kubernetes.io/projected/914abbd8-ce78-435e-b0ab-84357420340c-kube-api-access-fbts7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh\" (UID: \"914abbd8-ce78-435e-b0ab-84357420340c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh" Jul 10 16:44:36.292581 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.292537 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh" Jul 10 16:44:36.328079 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.328043 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6fm7\" (UniqueName: \"kubernetes.io/projected/4ac374f5-23a5-4e6c-9117-14523d4ec234-kube-api-access-t6fm7\") pod \"cert-manager-webhook-6486ffd997-2hcdv\" (UID: \"4ac374f5-23a5-4e6c-9117-14523d4ec234\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2hcdv" Jul 10 16:44:36.328079 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.328084 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ac374f5-23a5-4e6c-9117-14523d4ec234-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2hcdv\" (UID: \"4ac374f5-23a5-4e6c-9117-14523d4ec234\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2hcdv" Jul 10 16:44:36.428571 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.428540 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6fm7\" (UniqueName: \"kubernetes.io/projected/4ac374f5-23a5-4e6c-9117-14523d4ec234-kube-api-access-t6fm7\") pod \"cert-manager-webhook-6486ffd997-2hcdv\" (UID: \"4ac374f5-23a5-4e6c-9117-14523d4ec234\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2hcdv" Jul 10 16:44:36.428571 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.428574 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ac374f5-23a5-4e6c-9117-14523d4ec234-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2hcdv\" (UID: \"4ac374f5-23a5-4e6c-9117-14523d4ec234\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2hcdv" Jul 10 16:44:36.434618 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.434594 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh"] Jul 10 16:44:36.436320 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:44:36.436289 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914abbd8_ce78_435e_b0ab_84357420340c.slice/crio-ad50d5f4c85186eb1c096af692353e068d498ae6977a4898ad813354031fbfe3 WatchSource:0}: Error finding container ad50d5f4c85186eb1c096af692353e068d498ae6977a4898ad813354031fbfe3: Status 404 returned error can't find the container with id ad50d5f4c85186eb1c096af692353e068d498ae6977a4898ad813354031fbfe3 Jul 10 16:44:36.438851 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.438826 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ac374f5-23a5-4e6c-9117-14523d4ec234-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2hcdv\" (UID: \"4ac374f5-23a5-4e6c-9117-14523d4ec234\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2hcdv" Jul 10 16:44:36.438966 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.438858 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6fm7\" (UniqueName: \"kubernetes.io/projected/4ac374f5-23a5-4e6c-9117-14523d4ec234-kube-api-access-t6fm7\") pod \"cert-manager-webhook-6486ffd997-2hcdv\" (UID: \"4ac374f5-23a5-4e6c-9117-14523d4ec234\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2hcdv" Jul 10 16:44:36.495843 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.495780 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-2hcdv" Jul 10 16:44:36.623718 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.623693 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2hcdv"] Jul 10 16:44:36.626042 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:44:36.626014 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ac374f5_23a5_4e6c_9117_14523d4ec234.slice/crio-1460103640f7cf24a1a122b1a4a0853a79c1adcc03fb8ecaae79d006a2a4e061 WatchSource:0}: Error finding container 1460103640f7cf24a1a122b1a4a0853a79c1adcc03fb8ecaae79d006a2a4e061: Status 404 returned error can't find the container with id 1460103640f7cf24a1a122b1a4a0853a79c1adcc03fb8ecaae79d006a2a4e061 Jul 10 16:44:36.981517 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.981484 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-2hcdv" event={"ID":"4ac374f5-23a5-4e6c-9117-14523d4ec234","Type":"ContainerStarted","Data":"1460103640f7cf24a1a122b1a4a0853a79c1adcc03fb8ecaae79d006a2a4e061"} Jul 10 16:44:36.982960 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.982936 2585 generic.go:356] "Generic (PLEG): container finished" podID="914abbd8-ce78-435e-b0ab-84357420340c" containerID="134f6ceb1966ad90ec3bf4d17f2d2b14314750f1576b56786138edabcc868a3e" exitCode=0 Jul 10 16:44:36.983071 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.982993 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh" event={"ID":"914abbd8-ce78-435e-b0ab-84357420340c","Type":"ContainerDied","Data":"134f6ceb1966ad90ec3bf4d17f2d2b14314750f1576b56786138edabcc868a3e"} Jul 10 16:44:36.983071 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:36.983027 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh" event={"ID":"914abbd8-ce78-435e-b0ab-84357420340c","Type":"ContainerStarted","Data":"ad50d5f4c85186eb1c096af692353e068d498ae6977a4898ad813354031fbfe3"} Jul 10 16:44:39.999193 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:39.999099 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-2hcdv" event={"ID":"4ac374f5-23a5-4e6c-9117-14523d4ec234","Type":"ContainerStarted","Data":"75c7b502c5d6d33ee5693e81220f6b24bc50549329bd602d40fafb4348d819e9"} Jul 10 16:44:39.999193 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:39.999169 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-2hcdv" Jul 10 16:44:40.000952 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:40.000928 2585 generic.go:356] "Generic (PLEG): container finished" podID="914abbd8-ce78-435e-b0ab-84357420340c" containerID="1aecd1322384c68e18dd490710cf088a156b6d60bf4f97857eb0478d88ae8d3c" exitCode=0 Jul 10 16:44:40.001070 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:40.001003 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh" event={"ID":"914abbd8-ce78-435e-b0ab-84357420340c","Type":"ContainerDied","Data":"1aecd1322384c68e18dd490710cf088a156b6d60bf4f97857eb0478d88ae8d3c"} Jul 10 16:44:40.016551 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:40.016511 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-2hcdv" podStartSLOduration=0.965311435 podStartE2EDuration="4.016494557s" totalImagesPullingTime="3.051183122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:44:40.014969695 +0000 UTC m=+499.436846167" watchObservedRunningTime="2026-07-10 16:44:40.016494557 +0000 UTC m=+499.438371024" Jul 10 16:44:41.007467 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:41.007433 2585 generic.go:356] "Generic (PLEG): container finished" podID="914abbd8-ce78-435e-b0ab-84357420340c" containerID="eed578c0fa6d6ba60541a2966fc739fe1d6623c302933572fe43d7b848060cb4" exitCode=0 Jul 10 16:44:41.007871 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:41.007528 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh" event={"ID":"914abbd8-ce78-435e-b0ab-84357420340c","Type":"ContainerDied","Data":"eed578c0fa6d6ba60541a2966fc739fe1d6623c302933572fe43d7b848060cb4"} Jul 10 16:44:42.135550 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:42.135526 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh" Jul 10 16:44:42.274891 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:42.274802 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/914abbd8-ce78-435e-b0ab-84357420340c-util\") pod \"914abbd8-ce78-435e-b0ab-84357420340c\" (UID: \"914abbd8-ce78-435e-b0ab-84357420340c\") " Jul 10 16:44:42.274891 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:42.274836 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbts7\" (UniqueName: \"kubernetes.io/projected/914abbd8-ce78-435e-b0ab-84357420340c-kube-api-access-fbts7\") pod \"914abbd8-ce78-435e-b0ab-84357420340c\" (UID: \"914abbd8-ce78-435e-b0ab-84357420340c\") " Jul 10 16:44:42.274891 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:42.274877 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/914abbd8-ce78-435e-b0ab-84357420340c-bundle\") pod \"914abbd8-ce78-435e-b0ab-84357420340c\" (UID: \"914abbd8-ce78-435e-b0ab-84357420340c\") " Jul 10 16:44:42.275303 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:42.275266 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914abbd8-ce78-435e-b0ab-84357420340c-bundle" (OuterVolumeSpecName: "bundle") pod "914abbd8-ce78-435e-b0ab-84357420340c" (UID: "914abbd8-ce78-435e-b0ab-84357420340c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:44:42.276889 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:42.276867 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914abbd8-ce78-435e-b0ab-84357420340c-kube-api-access-fbts7" (OuterVolumeSpecName: "kube-api-access-fbts7") pod "914abbd8-ce78-435e-b0ab-84357420340c" (UID: "914abbd8-ce78-435e-b0ab-84357420340c"). InnerVolumeSpecName "kube-api-access-fbts7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:44:42.279114 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:42.279080 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914abbd8-ce78-435e-b0ab-84357420340c-util" (OuterVolumeSpecName: "util") pod "914abbd8-ce78-435e-b0ab-84357420340c" (UID: "914abbd8-ce78-435e-b0ab-84357420340c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:44:42.375879 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:42.375848 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/914abbd8-ce78-435e-b0ab-84357420340c-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:44:42.375879 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:42.375871 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/914abbd8-ce78-435e-b0ab-84357420340c-util\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:44:42.375879 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:42.375880 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbts7\" (UniqueName: \"kubernetes.io/projected/914abbd8-ce78-435e-b0ab-84357420340c-kube-api-access-fbts7\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:44:43.017220 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:43.017193 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh" Jul 10 16:44:43.017388 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:43.017190 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkvmnh" event={"ID":"914abbd8-ce78-435e-b0ab-84357420340c","Type":"ContainerDied","Data":"ad50d5f4c85186eb1c096af692353e068d498ae6977a4898ad813354031fbfe3"} Jul 10 16:44:43.017388 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:43.017316 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad50d5f4c85186eb1c096af692353e068d498ae6977a4898ad813354031fbfe3" Jul 10 16:44:46.009943 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:46.009908 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-2hcdv" Jul 10 16:44:56.527812 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.527769 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5"] Jul 10 16:44:56.528214 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.528110 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="914abbd8-ce78-435e-b0ab-84357420340c" containerName="util" Jul 10 16:44:56.528214 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.528123 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="914abbd8-ce78-435e-b0ab-84357420340c" containerName="util" Jul 10 16:44:56.528214 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.528132 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="914abbd8-ce78-435e-b0ab-84357420340c" containerName="pull" Jul 10 16:44:56.528214 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.528138 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="914abbd8-ce78-435e-b0ab-84357420340c" containerName="pull" Jul 10 16:44:56.528214 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.528156 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="914abbd8-ce78-435e-b0ab-84357420340c" containerName="extract" Jul 10 16:44:56.528214 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.528162 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="914abbd8-ce78-435e-b0ab-84357420340c" containerName="extract" Jul 10 16:44:56.528214 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.528206 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="914abbd8-ce78-435e-b0ab-84357420340c" containerName="extract" Jul 10 16:44:56.533244 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.533226 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5" Jul 10 16:44:56.536066 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.536039 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-25b9w\"" Jul 10 16:44:56.536289 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.536263 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 16:44:56.537500 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.537476 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 16:44:56.540139 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.540106 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5"] Jul 10 16:44:56.693917 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.693877 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77d70de7-bd95-4ecf-aa37-05bc1126ed51-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5\" (UID: \"77d70de7-bd95-4ecf-aa37-05bc1126ed51\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5" Jul 10 16:44:56.693917 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.693918 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc82v\" (UniqueName: \"kubernetes.io/projected/77d70de7-bd95-4ecf-aa37-05bc1126ed51-kube-api-access-nc82v\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5\" (UID: \"77d70de7-bd95-4ecf-aa37-05bc1126ed51\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5" Jul 10 16:44:56.694133 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.694003 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77d70de7-bd95-4ecf-aa37-05bc1126ed51-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5\" (UID: \"77d70de7-bd95-4ecf-aa37-05bc1126ed51\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5" Jul 10 16:44:56.795293 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.795187 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77d70de7-bd95-4ecf-aa37-05bc1126ed51-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5\" (UID: \"77d70de7-bd95-4ecf-aa37-05bc1126ed51\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5" Jul 10 16:44:56.795293 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.795232 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc82v\" (UniqueName: \"kubernetes.io/projected/77d70de7-bd95-4ecf-aa37-05bc1126ed51-kube-api-access-nc82v\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5\" (UID: \"77d70de7-bd95-4ecf-aa37-05bc1126ed51\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5" Jul 10 16:44:56.795566 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.795340 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77d70de7-bd95-4ecf-aa37-05bc1126ed51-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5\" (UID: \"77d70de7-bd95-4ecf-aa37-05bc1126ed51\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5" Jul 10 16:44:56.795686 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.795662 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77d70de7-bd95-4ecf-aa37-05bc1126ed51-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5\" (UID: \"77d70de7-bd95-4ecf-aa37-05bc1126ed51\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5" Jul 10 16:44:56.795772 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.795670 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77d70de7-bd95-4ecf-aa37-05bc1126ed51-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5\" (UID: \"77d70de7-bd95-4ecf-aa37-05bc1126ed51\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5" Jul 10 16:44:56.803243 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.803219 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc82v\" (UniqueName: \"kubernetes.io/projected/77d70de7-bd95-4ecf-aa37-05bc1126ed51-kube-api-access-nc82v\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5\" (UID: \"77d70de7-bd95-4ecf-aa37-05bc1126ed51\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5" Jul 10 16:44:56.843011 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.842972 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5" Jul 10 16:44:56.971357 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:56.971321 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5"] Jul 10 16:44:56.973413 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:44:56.973389 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d70de7_bd95_4ecf_aa37_05bc1126ed51.slice/crio-9c9ba3834a245d78d57a52e6c3a050efc3c3e63934836f61c2672f4bead5ffb7 WatchSource:0}: Error finding container 9c9ba3834a245d78d57a52e6c3a050efc3c3e63934836f61c2672f4bead5ffb7: Status 404 returned error can't find the container with id 9c9ba3834a245d78d57a52e6c3a050efc3c3e63934836f61c2672f4bead5ffb7 Jul 10 16:44:57.079401 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:57.079372 2585 generic.go:356] "Generic (PLEG): container finished" podID="77d70de7-bd95-4ecf-aa37-05bc1126ed51" containerID="2cafdaa965bf8d31545f5701bab929aa54e676d12f274a74d5ed0969c85c5588" exitCode=0 Jul 10 16:44:57.079543 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:57.079449 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5" event={"ID":"77d70de7-bd95-4ecf-aa37-05bc1126ed51","Type":"ContainerDied","Data":"2cafdaa965bf8d31545f5701bab929aa54e676d12f274a74d5ed0969c85c5588"} Jul 10 16:44:57.079543 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:57.079502 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5" event={"ID":"77d70de7-bd95-4ecf-aa37-05bc1126ed51","Type":"ContainerStarted","Data":"9c9ba3834a245d78d57a52e6c3a050efc3c3e63934836f61c2672f4bead5ffb7"} Jul 10 16:44:58.085342 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:58.085257 2585 generic.go:356] "Generic (PLEG): container finished" podID="77d70de7-bd95-4ecf-aa37-05bc1126ed51" containerID="cfe5fd36ffd99c715cb795ef3140c5276b16de529dac4762c05cf22b9cb2c803" exitCode=0 Jul 10 16:44:58.085703 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:58.085342 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5" event={"ID":"77d70de7-bd95-4ecf-aa37-05bc1126ed51","Type":"ContainerDied","Data":"cfe5fd36ffd99c715cb795ef3140c5276b16de529dac4762c05cf22b9cb2c803"} Jul 10 16:44:59.092144 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:59.092111 2585 generic.go:356] "Generic (PLEG): container finished" podID="77d70de7-bd95-4ecf-aa37-05bc1126ed51" containerID="331c63afa79e36301cdbb16ab9597e1dba36f88d0d5aff9276fdd134cdc36237" exitCode=0 Jul 10 16:44:59.092557 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:44:59.092190 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5" event={"ID":"77d70de7-bd95-4ecf-aa37-05bc1126ed51","Type":"ContainerDied","Data":"331c63afa79e36301cdbb16ab9597e1dba36f88d0d5aff9276fdd134cdc36237"} Jul 10 16:45:00.220106 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:00.220080 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5" Jul 10 16:45:00.223895 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:00.223869 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77d70de7-bd95-4ecf-aa37-05bc1126ed51-util\") pod \"77d70de7-bd95-4ecf-aa37-05bc1126ed51\" (UID: \"77d70de7-bd95-4ecf-aa37-05bc1126ed51\") " Jul 10 16:45:00.223895 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:00.223898 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77d70de7-bd95-4ecf-aa37-05bc1126ed51-bundle\") pod \"77d70de7-bd95-4ecf-aa37-05bc1126ed51\" (UID: \"77d70de7-bd95-4ecf-aa37-05bc1126ed51\") " Jul 10 16:45:00.224083 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:00.223932 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nc82v\" (UniqueName: \"kubernetes.io/projected/77d70de7-bd95-4ecf-aa37-05bc1126ed51-kube-api-access-nc82v\") pod \"77d70de7-bd95-4ecf-aa37-05bc1126ed51\" (UID: \"77d70de7-bd95-4ecf-aa37-05bc1126ed51\") " Jul 10 16:45:00.224777 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:00.224730 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d70de7-bd95-4ecf-aa37-05bc1126ed51-bundle" (OuterVolumeSpecName: "bundle") pod "77d70de7-bd95-4ecf-aa37-05bc1126ed51" (UID: "77d70de7-bd95-4ecf-aa37-05bc1126ed51"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:45:00.226031 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:00.226005 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d70de7-bd95-4ecf-aa37-05bc1126ed51-kube-api-access-nc82v" (OuterVolumeSpecName: "kube-api-access-nc82v") pod "77d70de7-bd95-4ecf-aa37-05bc1126ed51" (UID: "77d70de7-bd95-4ecf-aa37-05bc1126ed51"). InnerVolumeSpecName "kube-api-access-nc82v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:45:00.229549 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:00.229528 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d70de7-bd95-4ecf-aa37-05bc1126ed51-util" (OuterVolumeSpecName: "util") pod "77d70de7-bd95-4ecf-aa37-05bc1126ed51" (UID: "77d70de7-bd95-4ecf-aa37-05bc1126ed51"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:45:00.324900 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:00.324862 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc82v\" (UniqueName: \"kubernetes.io/projected/77d70de7-bd95-4ecf-aa37-05bc1126ed51-kube-api-access-nc82v\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:45:00.324900 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:00.324898 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77d70de7-bd95-4ecf-aa37-05bc1126ed51-util\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:45:00.325114 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:00.324912 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77d70de7-bd95-4ecf-aa37-05bc1126ed51-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:45:01.103502 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:01.103473 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5" Jul 10 16:45:01.103502 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:01.103491 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affqqn5" event={"ID":"77d70de7-bd95-4ecf-aa37-05bc1126ed51","Type":"ContainerDied","Data":"9c9ba3834a245d78d57a52e6c3a050efc3c3e63934836f61c2672f4bead5ffb7"} Jul 10 16:45:01.103703 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:01.103538 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c9ba3834a245d78d57a52e6c3a050efc3c3e63934836f61c2672f4bead5ffb7" Jul 10 16:45:09.311135 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.311104 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq"] Jul 10 16:45:09.311609 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.311408 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d70de7-bd95-4ecf-aa37-05bc1126ed51" containerName="extract" Jul 10 16:45:09.311609 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.311421 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d70de7-bd95-4ecf-aa37-05bc1126ed51" containerName="extract" Jul 10 16:45:09.311609 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.311430 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d70de7-bd95-4ecf-aa37-05bc1126ed51" containerName="pull" Jul 10 16:45:09.311609 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.311435 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d70de7-bd95-4ecf-aa37-05bc1126ed51" containerName="pull" Jul 10 16:45:09.311609 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.311445 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d70de7-bd95-4ecf-aa37-05bc1126ed51" containerName="util" Jul 10 16:45:09.311609 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.311450 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d70de7-bd95-4ecf-aa37-05bc1126ed51" containerName="util" Jul 10 16:45:09.311609 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.311501 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="77d70de7-bd95-4ecf-aa37-05bc1126ed51" containerName="extract" Jul 10 16:45:09.313982 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.313965 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:09.317025 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.317002 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 16:45:09.317142 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.317042 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 16:45:09.317248 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.317233 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-v48n5\"" Jul 10 16:45:09.317335 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.317242 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 16:45:09.317555 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.317543 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 16:45:09.333991 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.333963 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq"] Jul 10 16:45:09.391904 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.391875 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea27ada3-c6a4-4a07-947d-f92f52af0bb0-tmp\") pod \"opendatahub-operator-controller-manager-585db49996-v6pvq\" (UID: \"ea27ada3-c6a4-4a07-947d-f92f52af0bb0\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:09.392026 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.391916 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea27ada3-c6a4-4a07-947d-f92f52af0bb0-webhook-cert\") pod \"opendatahub-operator-controller-manager-585db49996-v6pvq\" (UID: \"ea27ada3-c6a4-4a07-947d-f92f52af0bb0\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:09.392026 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.391962 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ea27ada3-c6a4-4a07-947d-f92f52af0bb0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585db49996-v6pvq\" (UID: \"ea27ada3-c6a4-4a07-947d-f92f52af0bb0\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:09.392026 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.391984 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8k5d\" (UniqueName: \"kubernetes.io/projected/ea27ada3-c6a4-4a07-947d-f92f52af0bb0-kube-api-access-k8k5d\") pod \"opendatahub-operator-controller-manager-585db49996-v6pvq\" (UID: \"ea27ada3-c6a4-4a07-947d-f92f52af0bb0\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:09.392127 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.392068 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ea27ada3-c6a4-4a07-947d-f92f52af0bb0-manifests\") pod \"opendatahub-operator-controller-manager-585db49996-v6pvq\" (UID: \"ea27ada3-c6a4-4a07-947d-f92f52af0bb0\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:09.492715 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.492691 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ea27ada3-c6a4-4a07-947d-f92f52af0bb0-manifests\") pod \"opendatahub-operator-controller-manager-585db49996-v6pvq\" (UID: \"ea27ada3-c6a4-4a07-947d-f92f52af0bb0\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:09.492861 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.492724 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea27ada3-c6a4-4a07-947d-f92f52af0bb0-tmp\") pod \"opendatahub-operator-controller-manager-585db49996-v6pvq\" (UID: \"ea27ada3-c6a4-4a07-947d-f92f52af0bb0\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:09.492861 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.492761 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea27ada3-c6a4-4a07-947d-f92f52af0bb0-webhook-cert\") pod \"opendatahub-operator-controller-manager-585db49996-v6pvq\" (UID: \"ea27ada3-c6a4-4a07-947d-f92f52af0bb0\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:09.492861 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.492846 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ea27ada3-c6a4-4a07-947d-f92f52af0bb0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585db49996-v6pvq\" (UID: \"ea27ada3-c6a4-4a07-947d-f92f52af0bb0\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:09.493024 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.492873 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8k5d\" (UniqueName: \"kubernetes.io/projected/ea27ada3-c6a4-4a07-947d-f92f52af0bb0-kube-api-access-k8k5d\") pod \"opendatahub-operator-controller-manager-585db49996-v6pvq\" (UID: \"ea27ada3-c6a4-4a07-947d-f92f52af0bb0\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:09.493167 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.493126 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ea27ada3-c6a4-4a07-947d-f92f52af0bb0-manifests\") pod \"opendatahub-operator-controller-manager-585db49996-v6pvq\" (UID: \"ea27ada3-c6a4-4a07-947d-f92f52af0bb0\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:09.494857 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.494832 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea27ada3-c6a4-4a07-947d-f92f52af0bb0-tmp\") pod \"opendatahub-operator-controller-manager-585db49996-v6pvq\" (UID: \"ea27ada3-c6a4-4a07-947d-f92f52af0bb0\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:09.495099 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.495082 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ea27ada3-c6a4-4a07-947d-f92f52af0bb0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585db49996-v6pvq\" (UID: \"ea27ada3-c6a4-4a07-947d-f92f52af0bb0\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:09.495203 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.495184 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea27ada3-c6a4-4a07-947d-f92f52af0bb0-webhook-cert\") pod \"opendatahub-operator-controller-manager-585db49996-v6pvq\" (UID: \"ea27ada3-c6a4-4a07-947d-f92f52af0bb0\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:09.501993 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.501974 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8k5d\" (UniqueName: \"kubernetes.io/projected/ea27ada3-c6a4-4a07-947d-f92f52af0bb0-kube-api-access-k8k5d\") pod \"opendatahub-operator-controller-manager-585db49996-v6pvq\" (UID: \"ea27ada3-c6a4-4a07-947d-f92f52af0bb0\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:09.624216 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.624196 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:09.763297 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:09.763266 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq"] Jul 10 16:45:09.766447 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:45:09.766415 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea27ada3_c6a4_4a07_947d_f92f52af0bb0.slice/crio-44de47d9d2d5985b71779a0db466656f076c003722784ac2135c23cf41dea8be WatchSource:0}: Error finding container 44de47d9d2d5985b71779a0db466656f076c003722784ac2135c23cf41dea8be: Status 404 returned error can't find the container with id 44de47d9d2d5985b71779a0db466656f076c003722784ac2135c23cf41dea8be Jul 10 16:45:10.145990 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:10.145964 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" event={"ID":"ea27ada3-c6a4-4a07-947d-f92f52af0bb0","Type":"ContainerStarted","Data":"44de47d9d2d5985b71779a0db466656f076c003722784ac2135c23cf41dea8be"} Jul 10 16:45:13.165930 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:13.165863 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" event={"ID":"ea27ada3-c6a4-4a07-947d-f92f52af0bb0","Type":"ContainerStarted","Data":"c2e5d2f9a488e42e5f6024ed6343ed84e6ce3a7d17ca8584b6d1acbbd6d11401"} Jul 10 16:45:14.172006 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:14.171972 2585 generic.go:356] "Generic (PLEG): container finished" podID="ea27ada3-c6a4-4a07-947d-f92f52af0bb0" containerID="c2e5d2f9a488e42e5f6024ed6343ed84e6ce3a7d17ca8584b6d1acbbd6d11401" exitCode=0 Jul 10 16:45:14.172401 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:14.172019 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" event={"ID":"ea27ada3-c6a4-4a07-947d-f92f52af0bb0","Type":"ContainerDied","Data":"c2e5d2f9a488e42e5f6024ed6343ed84e6ce3a7d17ca8584b6d1acbbd6d11401"} Jul 10 16:45:15.178693 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:15.178650 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" event={"ID":"ea27ada3-c6a4-4a07-947d-f92f52af0bb0","Type":"ContainerStarted","Data":"ed97a066cb5848c3a5364a2adf2d79c2eaa79e26a9cd26cae31abb91935b1a76"} Jul 10 16:45:15.179116 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:15.178764 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:15.203582 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:15.203532 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" podStartSLOduration=1.645449781 podStartE2EDuration="6.203509616s" totalImagesPullingTime="3.391823818s" totalInitContainerRuntime="1.166236017s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:45:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:45:15.20133721 +0000 UTC m=+534.623213698" watchObservedRunningTime="2026-07-10 16:45:15.203509616 +0000 UTC m=+534.625386082" Jul 10 16:45:26.186294 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:26.186260 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-585db49996-v6pvq" Jul 10 16:45:27.941086 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:27.941050 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd"] Jul 10 16:45:27.955320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:27.955287 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" Jul 10 16:45:27.956224 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:27.956198 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd"] Jul 10 16:45:27.959761 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:27.959721 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 16:45:27.959866 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:27.959813 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 16:45:27.959866 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:27.959819 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cc924\"" Jul 10 16:45:27.960023 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:27.959886 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 16:45:27.960081 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:27.960030 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 16:45:27.960362 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:27.960342 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 16:45:28.053548 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:28.053512 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/990c9d67-17b9-4bda-a748-b4676da25ff0-manager-config\") pod \"lws-controller-manager-655c89f56d-g5rqd\" (UID: \"990c9d67-17b9-4bda-a748-b4676da25ff0\") " pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" Jul 10 16:45:28.053776 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:28.053574 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/990c9d67-17b9-4bda-a748-b4676da25ff0-cert\") pod \"lws-controller-manager-655c89f56d-g5rqd\" (UID: \"990c9d67-17b9-4bda-a748-b4676da25ff0\") " pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" Jul 10 16:45:28.053776 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:28.053599 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/990c9d67-17b9-4bda-a748-b4676da25ff0-metrics-cert\") pod \"lws-controller-manager-655c89f56d-g5rqd\" (UID: \"990c9d67-17b9-4bda-a748-b4676da25ff0\") " pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" Jul 10 16:45:28.053776 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:28.053667 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb8ll\" (UniqueName: \"kubernetes.io/projected/990c9d67-17b9-4bda-a748-b4676da25ff0-kube-api-access-xb8ll\") pod \"lws-controller-manager-655c89f56d-g5rqd\" (UID: \"990c9d67-17b9-4bda-a748-b4676da25ff0\") " pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" Jul 10 16:45:28.154757 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:28.154705 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb8ll\" (UniqueName: \"kubernetes.io/projected/990c9d67-17b9-4bda-a748-b4676da25ff0-kube-api-access-xb8ll\") pod \"lws-controller-manager-655c89f56d-g5rqd\" (UID: \"990c9d67-17b9-4bda-a748-b4676da25ff0\") " pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" Jul 10 16:45:28.154928 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:28.154779 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/990c9d67-17b9-4bda-a748-b4676da25ff0-manager-config\") pod \"lws-controller-manager-655c89f56d-g5rqd\" (UID: \"990c9d67-17b9-4bda-a748-b4676da25ff0\") " pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" Jul 10 16:45:28.154974 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:28.154947 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/990c9d67-17b9-4bda-a748-b4676da25ff0-cert\") pod \"lws-controller-manager-655c89f56d-g5rqd\" (UID: \"990c9d67-17b9-4bda-a748-b4676da25ff0\") " pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" Jul 10 16:45:28.155009 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:28.154988 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/990c9d67-17b9-4bda-a748-b4676da25ff0-metrics-cert\") pod \"lws-controller-manager-655c89f56d-g5rqd\" (UID: \"990c9d67-17b9-4bda-a748-b4676da25ff0\") " pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" Jul 10 16:45:28.155503 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:28.155478 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/990c9d67-17b9-4bda-a748-b4676da25ff0-manager-config\") pod \"lws-controller-manager-655c89f56d-g5rqd\" (UID: \"990c9d67-17b9-4bda-a748-b4676da25ff0\") " pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" Jul 10 16:45:28.157523 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:28.157499 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/990c9d67-17b9-4bda-a748-b4676da25ff0-metrics-cert\") pod \"lws-controller-manager-655c89f56d-g5rqd\" (UID: \"990c9d67-17b9-4bda-a748-b4676da25ff0\") " pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" Jul 10 16:45:28.157635 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:28.157616 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/990c9d67-17b9-4bda-a748-b4676da25ff0-cert\") pod \"lws-controller-manager-655c89f56d-g5rqd\" (UID: \"990c9d67-17b9-4bda-a748-b4676da25ff0\") " pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" Jul 10 16:45:28.170195 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:28.170170 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb8ll\" (UniqueName: \"kubernetes.io/projected/990c9d67-17b9-4bda-a748-b4676da25ff0-kube-api-access-xb8ll\") pod \"lws-controller-manager-655c89f56d-g5rqd\" (UID: \"990c9d67-17b9-4bda-a748-b4676da25ff0\") " pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" Jul 10 16:45:28.266177 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:28.266093 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" Jul 10 16:45:28.461077 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:28.461052 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd"] Jul 10 16:45:28.463669 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:45:28.463643 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990c9d67_17b9_4bda_a748_b4676da25ff0.slice/crio-fb01032eba10214c9b2fe170471890f8e91dc73cc8080caa6143df6db1ab66e6 WatchSource:0}: Error finding container fb01032eba10214c9b2fe170471890f8e91dc73cc8080caa6143df6db1ab66e6: Status 404 returned error can't find the container with id fb01032eba10214c9b2fe170471890f8e91dc73cc8080caa6143df6db1ab66e6 Jul 10 16:45:29.245493 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:29.245448 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" event={"ID":"990c9d67-17b9-4bda-a748-b4676da25ff0","Type":"ContainerStarted","Data":"fb01032eba10214c9b2fe170471890f8e91dc73cc8080caa6143df6db1ab66e6"} Jul 10 16:45:39.297160 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:39.297124 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" event={"ID":"990c9d67-17b9-4bda-a748-b4676da25ff0","Type":"ContainerStarted","Data":"67ebc3e21a5da15f06a670f7f673a3992bbaf51de1583f4085b71a362c1b9c95"} Jul 10 16:45:39.297553 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:39.297266 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" Jul 10 16:45:39.321065 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:39.321011 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" podStartSLOduration=2.392013528 podStartE2EDuration="12.320996147s" totalImagesPullingTime="9.928982619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:45:39.318796041 +0000 UTC m=+558.740672527" watchObservedRunningTime="2026-07-10 16:45:39.320996147 +0000 UTC m=+558.742872614" Jul 10 16:45:48.546820 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.545008 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8"] Jul 10 16:45:48.550144 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.550120 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.552904 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.552882 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 16:45:48.553012 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.552884 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-n4xzh\"" Jul 10 16:45:48.559369 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.559344 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8"] Jul 10 16:45:48.724570 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.724536 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0909b02-35a0-4e85-9c1f-a43508d37f0a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.724710 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.724576 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxzdt\" (UniqueName: \"kubernetes.io/projected/d0909b02-35a0-4e85-9c1f-a43508d37f0a-kube-api-access-qxzdt\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.724710 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.724601 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0909b02-35a0-4e85-9c1f-a43508d37f0a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.724710 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.724626 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0909b02-35a0-4e85-9c1f-a43508d37f0a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.724710 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.724687 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0909b02-35a0-4e85-9c1f-a43508d37f0a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.724869 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.724709 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0909b02-35a0-4e85-9c1f-a43508d37f0a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.724869 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.724785 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0909b02-35a0-4e85-9c1f-a43508d37f0a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.724869 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.724818 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0909b02-35a0-4e85-9c1f-a43508d37f0a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.724869 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.724838 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0909b02-35a0-4e85-9c1f-a43508d37f0a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.825644 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.825572 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0909b02-35a0-4e85-9c1f-a43508d37f0a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.825644 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.825609 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxzdt\" (UniqueName: \"kubernetes.io/projected/d0909b02-35a0-4e85-9c1f-a43508d37f0a-kube-api-access-qxzdt\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.825644 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.825634 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0909b02-35a0-4e85-9c1f-a43508d37f0a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.825905 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.825660 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0909b02-35a0-4e85-9c1f-a43508d37f0a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.825905 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.825865 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0909b02-35a0-4e85-9c1f-a43508d37f0a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.826016 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.825925 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0909b02-35a0-4e85-9c1f-a43508d37f0a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.826016 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.825965 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0909b02-35a0-4e85-9c1f-a43508d37f0a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.826016 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.825995 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0909b02-35a0-4e85-9c1f-a43508d37f0a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.826154 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.826026 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0909b02-35a0-4e85-9c1f-a43508d37f0a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.826154 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.826045 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0909b02-35a0-4e85-9c1f-a43508d37f0a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.826154 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.826111 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0909b02-35a0-4e85-9c1f-a43508d37f0a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.826259 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.826221 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0909b02-35a0-4e85-9c1f-a43508d37f0a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.826390 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.826368 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0909b02-35a0-4e85-9c1f-a43508d37f0a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.826797 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.826775 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0909b02-35a0-4e85-9c1f-a43508d37f0a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.828128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.828110 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0909b02-35a0-4e85-9c1f-a43508d37f0a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.828209 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.828188 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0909b02-35a0-4e85-9c1f-a43508d37f0a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.833484 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.833466 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0909b02-35a0-4e85-9c1f-a43508d37f0a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.833730 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.833713 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxzdt\" (UniqueName: \"kubernetes.io/projected/d0909b02-35a0-4e85-9c1f-a43508d37f0a-kube-api-access-qxzdt\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hdmv8\" (UID: \"d0909b02-35a0-4e85-9c1f-a43508d37f0a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:48.861717 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:48.861692 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:49.005075 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:49.005051 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8"] Jul 10 16:45:49.007612 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:45:49.007575 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0909b02_35a0_4e85_9c1f_a43508d37f0a.slice/crio-fa1cd337b654652496643c08f285d19868f74d6fb486aefb2414f71398ac1801 WatchSource:0}: Error finding container fa1cd337b654652496643c08f285d19868f74d6fb486aefb2414f71398ac1801: Status 404 returned error can't find the container with id fa1cd337b654652496643c08f285d19868f74d6fb486aefb2414f71398ac1801 Jul 10 16:45:49.342531 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:49.342497 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" event={"ID":"d0909b02-35a0-4e85-9c1f-a43508d37f0a","Type":"ContainerStarted","Data":"fa1cd337b654652496643c08f285d19868f74d6fb486aefb2414f71398ac1801"} Jul 10 16:45:50.304957 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:50.304924 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-655c89f56d-g5rqd" Jul 10 16:45:51.881908 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:51.881871 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 16:45:51.882187 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:51.881939 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 16:45:51.882187 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:51.881970 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 16:45:52.361944 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:52.361897 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" event={"ID":"d0909b02-35a0-4e85-9c1f-a43508d37f0a","Type":"ContainerStarted","Data":"ba21e92e9894b7f9d174706d3d9421b9ce7de06195373641aa3e00af8db4c1b9"} Jul 10 16:45:52.383178 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:52.383131 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" podStartSLOduration=1.5116220500000002 podStartE2EDuration="4.383117577s" totalImagesPullingTime="2.871495527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:45:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:45:52.381247595 +0000 UTC m=+571.803124086" watchObservedRunningTime="2026-07-10 16:45:52.383117577 +0000 UTC m=+571.804994043" Jul 10 16:45:52.862098 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:52.862066 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:52.866417 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:52.866391 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:53.366600 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:53.366571 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:45:53.367375 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:45:53.367356 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hdmv8" Jul 10 16:46:20.988887 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:20.988860 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 16:46:20.989635 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:20.989618 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 16:46:30.145938 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:30.145899 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vkw7g"] Jul 10 16:46:30.154262 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:30.154237 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vkw7g" Jul 10 16:46:30.157467 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:30.157404 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vkw7g"] Jul 10 16:46:30.158169 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:30.157767 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 16:46:30.158718 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:30.158547 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-dg4m7\"" Jul 10 16:46:30.158718 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:30.158699 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 16:46:30.255898 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:30.255873 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk5hw\" (UniqueName: \"kubernetes.io/projected/dbbd123e-3b2b-4090-8ad6-60d6d4877d66-kube-api-access-pk5hw\") pod \"kuadrant-operator-catalog-vkw7g\" (UID: \"dbbd123e-3b2b-4090-8ad6-60d6d4877d66\") " pod="kuadrant-system/kuadrant-operator-catalog-vkw7g" Jul 10 16:46:30.356537 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:30.356511 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk5hw\" (UniqueName: \"kubernetes.io/projected/dbbd123e-3b2b-4090-8ad6-60d6d4877d66-kube-api-access-pk5hw\") pod \"kuadrant-operator-catalog-vkw7g\" (UID: \"dbbd123e-3b2b-4090-8ad6-60d6d4877d66\") " pod="kuadrant-system/kuadrant-operator-catalog-vkw7g" Jul 10 16:46:30.365288 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:30.365262 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk5hw\" (UniqueName: \"kubernetes.io/projected/dbbd123e-3b2b-4090-8ad6-60d6d4877d66-kube-api-access-pk5hw\") pod \"kuadrant-operator-catalog-vkw7g\" (UID: \"dbbd123e-3b2b-4090-8ad6-60d6d4877d66\") " pod="kuadrant-system/kuadrant-operator-catalog-vkw7g" Jul 10 16:46:30.466721 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:30.466657 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vkw7g" Jul 10 16:46:30.534005 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:30.533983 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vkw7g"] Jul 10 16:46:30.612561 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:30.612537 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vkw7g"] Jul 10 16:46:30.614857 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:46:30.614831 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbbd123e_3b2b_4090_8ad6_60d6d4877d66.slice/crio-5b18501b0332ea7813900b1c54deb81340ef23405b3ad2abe12bc538f9c96403 WatchSource:0}: Error finding container 5b18501b0332ea7813900b1c54deb81340ef23405b3ad2abe12bc538f9c96403: Status 404 returned error can't find the container with id 5b18501b0332ea7813900b1c54deb81340ef23405b3ad2abe12bc538f9c96403 Jul 10 16:46:30.731731 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:30.731671 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sn6r9"] Jul 10 16:46:30.736475 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:30.736459 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sn6r9" Jul 10 16:46:30.743102 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:30.743079 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sn6r9"] Jul 10 16:46:30.861260 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:30.861236 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfmd5\" (UniqueName: \"kubernetes.io/projected/2958cd40-3889-4149-86ff-70d50b439824-kube-api-access-mfmd5\") pod \"kuadrant-operator-catalog-sn6r9\" (UID: \"2958cd40-3889-4149-86ff-70d50b439824\") " pod="kuadrant-system/kuadrant-operator-catalog-sn6r9" Jul 10 16:46:30.962461 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:30.962436 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfmd5\" (UniqueName: \"kubernetes.io/projected/2958cd40-3889-4149-86ff-70d50b439824-kube-api-access-mfmd5\") pod \"kuadrant-operator-catalog-sn6r9\" (UID: \"2958cd40-3889-4149-86ff-70d50b439824\") " pod="kuadrant-system/kuadrant-operator-catalog-sn6r9" Jul 10 16:46:30.970817 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:30.970791 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfmd5\" (UniqueName: \"kubernetes.io/projected/2958cd40-3889-4149-86ff-70d50b439824-kube-api-access-mfmd5\") pod \"kuadrant-operator-catalog-sn6r9\" (UID: \"2958cd40-3889-4149-86ff-70d50b439824\") " pod="kuadrant-system/kuadrant-operator-catalog-sn6r9" Jul 10 16:46:31.046139 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:31.046072 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sn6r9" Jul 10 16:46:31.180117 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:31.180094 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sn6r9"] Jul 10 16:46:31.187735 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:46:31.187709 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2958cd40_3889_4149_86ff_70d50b439824.slice/crio-99bc2d90eb55f6bb6d5b3936a9da4e77b883c629c1f60af5627a35e2ddb9a34d WatchSource:0}: Error finding container 99bc2d90eb55f6bb6d5b3936a9da4e77b883c629c1f60af5627a35e2ddb9a34d: Status 404 returned error can't find the container with id 99bc2d90eb55f6bb6d5b3936a9da4e77b883c629c1f60af5627a35e2ddb9a34d Jul 10 16:46:31.532988 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:31.532951 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sn6r9" event={"ID":"2958cd40-3889-4149-86ff-70d50b439824","Type":"ContainerStarted","Data":"99bc2d90eb55f6bb6d5b3936a9da4e77b883c629c1f60af5627a35e2ddb9a34d"} Jul 10 16:46:31.534110 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:31.534084 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vkw7g" event={"ID":"dbbd123e-3b2b-4090-8ad6-60d6d4877d66","Type":"ContainerStarted","Data":"5b18501b0332ea7813900b1c54deb81340ef23405b3ad2abe12bc538f9c96403"} Jul 10 16:46:33.545572 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:33.545532 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sn6r9" event={"ID":"2958cd40-3889-4149-86ff-70d50b439824","Type":"ContainerStarted","Data":"7a5e9f8c628fd0154a0e4b0fbcc17602b638760a68f216adb6c854fa3f97b4af"} Jul 10 16:46:33.547099 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:33.547074 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vkw7g" event={"ID":"dbbd123e-3b2b-4090-8ad6-60d6d4877d66","Type":"ContainerStarted","Data":"ff898e5b4d9796d41ecca162f0a7dad2c19516409467186bd4ca39c0cb44f53e"} Jul 10 16:46:33.547222 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:33.547110 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-vkw7g" podUID="dbbd123e-3b2b-4090-8ad6-60d6d4877d66" containerName="registry-server" containerID="cri-o://ff898e5b4d9796d41ecca162f0a7dad2c19516409467186bd4ca39c0cb44f53e" gracePeriod=2 Jul 10 16:46:33.562821 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:33.562776 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-sn6r9" podStartSLOduration=2.06456397 podStartE2EDuration="3.562758841s" totalImagesPullingTime="1.498194871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:46:33.562584972 +0000 UTC m=+612.984461482" watchObservedRunningTime="2026-07-10 16:46:33.562758841 +0000 UTC m=+612.984635299" Jul 10 16:46:33.582900 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:33.582863 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-vkw7g" podStartSLOduration=1.514339413 podStartE2EDuration="3.582850265s" totalImagesPullingTime="2.068510852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:46:33.580829067 +0000 UTC m=+613.002705562" watchObservedRunningTime="2026-07-10 16:46:33.582850265 +0000 UTC m=+613.004726732" Jul 10 16:46:33.793185 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:33.793162 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vkw7g" Jul 10 16:46:33.887898 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:33.887873 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pk5hw\" (UniqueName: \"kubernetes.io/projected/dbbd123e-3b2b-4090-8ad6-60d6d4877d66-kube-api-access-pk5hw\") pod \"dbbd123e-3b2b-4090-8ad6-60d6d4877d66\" (UID: \"dbbd123e-3b2b-4090-8ad6-60d6d4877d66\") " Jul 10 16:46:33.889908 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:33.889884 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbbd123e-3b2b-4090-8ad6-60d6d4877d66-kube-api-access-pk5hw" (OuterVolumeSpecName: "kube-api-access-pk5hw") pod "dbbd123e-3b2b-4090-8ad6-60d6d4877d66" (UID: "dbbd123e-3b2b-4090-8ad6-60d6d4877d66"). InnerVolumeSpecName "kube-api-access-pk5hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:46:33.989353 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:33.989331 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk5hw\" (UniqueName: \"kubernetes.io/projected/dbbd123e-3b2b-4090-8ad6-60d6d4877d66-kube-api-access-pk5hw\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:46:34.553520 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.553485 2585 generic.go:356] "Generic (PLEG): container finished" podID="dbbd123e-3b2b-4090-8ad6-60d6d4877d66" containerID="ff898e5b4d9796d41ecca162f0a7dad2c19516409467186bd4ca39c0cb44f53e" exitCode=0 Jul 10 16:46:34.553900 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.553533 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vkw7g" Jul 10 16:46:34.553900 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.553569 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vkw7g" event={"ID":"dbbd123e-3b2b-4090-8ad6-60d6d4877d66","Type":"ContainerDied","Data":"ff898e5b4d9796d41ecca162f0a7dad2c19516409467186bd4ca39c0cb44f53e"} Jul 10 16:46:34.553900 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.553620 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vkw7g" event={"ID":"dbbd123e-3b2b-4090-8ad6-60d6d4877d66","Type":"ContainerDied","Data":"5b18501b0332ea7813900b1c54deb81340ef23405b3ad2abe12bc538f9c96403"} Jul 10 16:46:34.553900 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.553642 2585 scope.go:117] "RemoveContainer" containerID="ff898e5b4d9796d41ecca162f0a7dad2c19516409467186bd4ca39c0cb44f53e" Jul 10 16:46:34.562812 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.562797 2585 scope.go:117] "RemoveContainer" containerID="ff898e5b4d9796d41ecca162f0a7dad2c19516409467186bd4ca39c0cb44f53e" Jul 10 16:46:34.563058 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:46:34.563040 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff898e5b4d9796d41ecca162f0a7dad2c19516409467186bd4ca39c0cb44f53e\": container with ID starting with ff898e5b4d9796d41ecca162f0a7dad2c19516409467186bd4ca39c0cb44f53e not found: ID does not exist" containerID="ff898e5b4d9796d41ecca162f0a7dad2c19516409467186bd4ca39c0cb44f53e" Jul 10 16:46:34.563106 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.563068 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff898e5b4d9796d41ecca162f0a7dad2c19516409467186bd4ca39c0cb44f53e"} err="failed to get container status \"ff898e5b4d9796d41ecca162f0a7dad2c19516409467186bd4ca39c0cb44f53e\": rpc error: code = NotFound desc = could not find container \"ff898e5b4d9796d41ecca162f0a7dad2c19516409467186bd4ca39c0cb44f53e\": container with ID starting with ff898e5b4d9796d41ecca162f0a7dad2c19516409467186bd4ca39c0cb44f53e not found: ID does not exist" Jul 10 16:46:34.574546 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.574526 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vkw7g"] Jul 10 16:46:34.578387 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.578366 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vkw7g"] Jul 10 16:46:34.731084 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.731058 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54cd88dfbf-sw796"] Jul 10 16:46:34.731344 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.731332 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbbd123e-3b2b-4090-8ad6-60d6d4877d66" containerName="registry-server" Jul 10 16:46:34.731386 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.731345 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbbd123e-3b2b-4090-8ad6-60d6d4877d66" containerName="registry-server" Jul 10 16:46:34.731420 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.731410 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbbd123e-3b2b-4090-8ad6-60d6d4877d66" containerName="registry-server" Jul 10 16:46:34.735909 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.735887 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.738715 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.738697 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-b49fg\"" Jul 10 16:46:34.738832 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.738735 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 16:46:34.738964 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.738951 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 16:46:34.739025 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.738969 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 16:46:34.739025 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.739016 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 16:46:34.739133 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.739039 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 16:46:34.739133 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.739128 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 16:46:34.739229 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.739170 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 16:46:34.744887 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.744859 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 16:46:34.749409 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.749389 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54cd88dfbf-sw796"] Jul 10 16:46:34.896443 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.896415 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04deca7a-3598-40c3-9fe6-c247f1712840-console-config\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.896532 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.896448 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04deca7a-3598-40c3-9fe6-c247f1712840-service-ca\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.896532 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.896464 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcp8w\" (UniqueName: \"kubernetes.io/projected/04deca7a-3598-40c3-9fe6-c247f1712840-kube-api-access-bcp8w\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.896532 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.896482 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04deca7a-3598-40c3-9fe6-c247f1712840-console-oauth-config\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.896670 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.896589 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04deca7a-3598-40c3-9fe6-c247f1712840-trusted-ca-bundle\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.896670 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.896619 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04deca7a-3598-40c3-9fe6-c247f1712840-oauth-serving-cert\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.896670 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.896641 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04deca7a-3598-40c3-9fe6-c247f1712840-console-serving-cert\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.997367 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.997342 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04deca7a-3598-40c3-9fe6-c247f1712840-trusted-ca-bundle\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.997452 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.997373 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04deca7a-3598-40c3-9fe6-c247f1712840-oauth-serving-cert\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.997452 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.997393 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04deca7a-3598-40c3-9fe6-c247f1712840-console-serving-cert\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.997452 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.997411 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04deca7a-3598-40c3-9fe6-c247f1712840-console-config\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.997452 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.997431 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04deca7a-3598-40c3-9fe6-c247f1712840-service-ca\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.997452 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.997447 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcp8w\" (UniqueName: \"kubernetes.io/projected/04deca7a-3598-40c3-9fe6-c247f1712840-kube-api-access-bcp8w\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.997688 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.997474 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04deca7a-3598-40c3-9fe6-c247f1712840-console-oauth-config\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.998195 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.998175 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04deca7a-3598-40c3-9fe6-c247f1712840-oauth-serving-cert\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.998289 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.998202 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04deca7a-3598-40c3-9fe6-c247f1712840-service-ca\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.998289 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.998213 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04deca7a-3598-40c3-9fe6-c247f1712840-console-config\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.998409 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.998293 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04deca7a-3598-40c3-9fe6-c247f1712840-trusted-ca-bundle\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.999911 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.999893 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04deca7a-3598-40c3-9fe6-c247f1712840-console-oauth-config\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:34.999985 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:34.999965 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04deca7a-3598-40c3-9fe6-c247f1712840-console-serving-cert\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:35.006046 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:35.006021 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcp8w\" (UniqueName: \"kubernetes.io/projected/04deca7a-3598-40c3-9fe6-c247f1712840-kube-api-access-bcp8w\") pod \"console-54cd88dfbf-sw796\" (UID: \"04deca7a-3598-40c3-9fe6-c247f1712840\") " pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:35.046029 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:35.046007 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:35.174652 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:35.174630 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54cd88dfbf-sw796"] Jul 10 16:46:35.177065 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:46:35.177031 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04deca7a_3598_40c3_9fe6_c247f1712840.slice/crio-50120d571b24dc54a871492e1b4ff73689199b33c5233571b38b3734dbd69f14 WatchSource:0}: Error finding container 50120d571b24dc54a871492e1b4ff73689199b33c5233571b38b3734dbd69f14: Status 404 returned error can't find the container with id 50120d571b24dc54a871492e1b4ff73689199b33c5233571b38b3734dbd69f14 Jul 10 16:46:35.325594 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:35.325564 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbbd123e-3b2b-4090-8ad6-60d6d4877d66" path="/var/lib/kubelet/pods/dbbd123e-3b2b-4090-8ad6-60d6d4877d66/volumes" Jul 10 16:46:35.564113 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:35.564036 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cd88dfbf-sw796" event={"ID":"04deca7a-3598-40c3-9fe6-c247f1712840","Type":"ContainerStarted","Data":"8a146a9b527f2b9110c982b364fae4023d59b2111f0dbf64310561a7c1fe309b"} Jul 10 16:46:35.564113 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:35.564104 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cd88dfbf-sw796" event={"ID":"04deca7a-3598-40c3-9fe6-c247f1712840","Type":"ContainerStarted","Data":"50120d571b24dc54a871492e1b4ff73689199b33c5233571b38b3734dbd69f14"} Jul 10 16:46:35.584883 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:35.584838 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54cd88dfbf-sw796" podStartSLOduration=1.584823846 podStartE2EDuration="1.584823846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:46:35.582297873 +0000 UTC m=+615.004174352" watchObservedRunningTime="2026-07-10 16:46:35.584823846 +0000 UTC m=+615.006700312" Jul 10 16:46:41.046716 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:41.046680 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-sn6r9" Jul 10 16:46:41.047096 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:41.046740 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-sn6r9" Jul 10 16:46:41.069641 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:41.069612 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-sn6r9" Jul 10 16:46:41.612322 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:41.612294 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-sn6r9" Jul 10 16:46:45.046701 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:45.046626 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:45.046701 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:45.046678 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:45.051152 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:45.051128 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:45.612394 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:45.612361 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54cd88dfbf-sw796" Jul 10 16:46:45.946520 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:45.946446 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45"] Jul 10 16:46:45.949851 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:45.949831 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45" Jul 10 16:46:45.952583 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:45.952567 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pzmp4\"" Jul 10 16:46:45.958345 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:45.958323 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45"] Jul 10 16:46:46.078807 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.078777 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b235db81-1b87-4a7c-8a5a-547ce8ca845a-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45\" (UID: \"b235db81-1b87-4a7c-8a5a-547ce8ca845a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45" Jul 10 16:46:46.079141 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.078814 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b235db81-1b87-4a7c-8a5a-547ce8ca845a-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45\" (UID: \"b235db81-1b87-4a7c-8a5a-547ce8ca845a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45" Jul 10 16:46:46.079141 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.078926 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bffp2\" (UniqueName: \"kubernetes.io/projected/b235db81-1b87-4a7c-8a5a-547ce8ca845a-kube-api-access-bffp2\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45\" (UID: \"b235db81-1b87-4a7c-8a5a-547ce8ca845a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45" Jul 10 16:46:46.179221 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.179198 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b235db81-1b87-4a7c-8a5a-547ce8ca845a-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45\" (UID: \"b235db81-1b87-4a7c-8a5a-547ce8ca845a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45" Jul 10 16:46:46.179348 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.179233 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b235db81-1b87-4a7c-8a5a-547ce8ca845a-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45\" (UID: \"b235db81-1b87-4a7c-8a5a-547ce8ca845a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45" Jul 10 16:46:46.179348 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.179302 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bffp2\" (UniqueName: \"kubernetes.io/projected/b235db81-1b87-4a7c-8a5a-547ce8ca845a-kube-api-access-bffp2\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45\" (UID: \"b235db81-1b87-4a7c-8a5a-547ce8ca845a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45" Jul 10 16:46:46.179545 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.179524 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b235db81-1b87-4a7c-8a5a-547ce8ca845a-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45\" (UID: \"b235db81-1b87-4a7c-8a5a-547ce8ca845a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45" Jul 10 16:46:46.179606 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.179568 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b235db81-1b87-4a7c-8a5a-547ce8ca845a-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45\" (UID: \"b235db81-1b87-4a7c-8a5a-547ce8ca845a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45" Jul 10 16:46:46.188215 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.188192 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bffp2\" (UniqueName: \"kubernetes.io/projected/b235db81-1b87-4a7c-8a5a-547ce8ca845a-kube-api-access-bffp2\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45\" (UID: \"b235db81-1b87-4a7c-8a5a-547ce8ca845a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45" Jul 10 16:46:46.259973 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.259927 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45" Jul 10 16:46:46.386875 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.386848 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45"] Jul 10 16:46:46.388453 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:46:46.388422 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb235db81_1b87_4a7c_8a5a_547ce8ca845a.slice/crio-9e32520b85c0b82f0505fe5d73df0e0acd23617846031d9ecb8ab04b88769d7a WatchSource:0}: Error finding container 9e32520b85c0b82f0505fe5d73df0e0acd23617846031d9ecb8ab04b88769d7a: Status 404 returned error can't find the container with id 9e32520b85c0b82f0505fe5d73df0e0acd23617846031d9ecb8ab04b88769d7a Jul 10 16:46:46.614766 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.614720 2585 generic.go:356] "Generic (PLEG): container finished" podID="b235db81-1b87-4a7c-8a5a-547ce8ca845a" containerID="4ec94f122286c982dbe672efa0ce4aa10ca393c34ae4a9ed23c0d0764f2007e4" exitCode=0 Jul 10 16:46:46.614875 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.614806 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45" event={"ID":"b235db81-1b87-4a7c-8a5a-547ce8ca845a","Type":"ContainerDied","Data":"4ec94f122286c982dbe672efa0ce4aa10ca393c34ae4a9ed23c0d0764f2007e4"} Jul 10 16:46:46.614921 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.614871 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45" event={"ID":"b235db81-1b87-4a7c-8a5a-547ce8ca845a","Type":"ContainerStarted","Data":"9e32520b85c0b82f0505fe5d73df0e0acd23617846031d9ecb8ab04b88769d7a"} Jul 10 16:46:46.745307 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.745278 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv"] Jul 10 16:46:46.748775 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.748735 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv" Jul 10 16:46:46.757358 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.757337 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv"] Jul 10 16:46:46.886249 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.886197 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c984eaa6-2d5a-41e1-8b6c-1def3e97c81a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv\" (UID: \"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv" Jul 10 16:46:46.886249 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.886227 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c984eaa6-2d5a-41e1-8b6c-1def3e97c81a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv\" (UID: \"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv" Jul 10 16:46:46.886249 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.886248 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqspr\" (UniqueName: \"kubernetes.io/projected/c984eaa6-2d5a-41e1-8b6c-1def3e97c81a-kube-api-access-lqspr\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv\" (UID: \"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv" Jul 10 16:46:46.986641 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.986619 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c984eaa6-2d5a-41e1-8b6c-1def3e97c81a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv\" (UID: \"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv" Jul 10 16:46:46.986769 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.986647 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c984eaa6-2d5a-41e1-8b6c-1def3e97c81a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv\" (UID: \"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv" Jul 10 16:46:46.986769 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.986671 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqspr\" (UniqueName: \"kubernetes.io/projected/c984eaa6-2d5a-41e1-8b6c-1def3e97c81a-kube-api-access-lqspr\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv\" (UID: \"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv" Jul 10 16:46:46.986974 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.986953 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c984eaa6-2d5a-41e1-8b6c-1def3e97c81a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv\" (UID: \"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv" Jul 10 16:46:46.987040 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.987017 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c984eaa6-2d5a-41e1-8b6c-1def3e97c81a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv\" (UID: \"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv" Jul 10 16:46:46.994907 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:46.994886 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqspr\" (UniqueName: \"kubernetes.io/projected/c984eaa6-2d5a-41e1-8b6c-1def3e97c81a-kube-api-access-lqspr\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv\" (UID: \"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv" Jul 10 16:46:47.057986 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.057960 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv" Jul 10 16:46:47.181287 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.181263 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv"] Jul 10 16:46:47.182586 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:46:47.182558 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc984eaa6_2d5a_41e1_8b6c_1def3e97c81a.slice/crio-1349f3fa9b7f5fd89271d844f2fa4567b13ebe3e288017e80d23b2e6d3d91e0d WatchSource:0}: Error finding container 1349f3fa9b7f5fd89271d844f2fa4567b13ebe3e288017e80d23b2e6d3d91e0d: Status 404 returned error can't find the container with id 1349f3fa9b7f5fd89271d844f2fa4567b13ebe3e288017e80d23b2e6d3d91e0d Jul 10 16:46:47.340088 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.340036 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx"] Jul 10 16:46:47.345533 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.345510 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" Jul 10 16:46:47.351738 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.351716 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx"] Jul 10 16:46:47.491107 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.491085 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvfwp\" (UniqueName: \"kubernetes.io/projected/60523f32-9f6f-4937-99c2-fc7aa9d96655-kube-api-access-xvfwp\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx\" (UID: \"60523f32-9f6f-4937-99c2-fc7aa9d96655\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" Jul 10 16:46:47.491174 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.491124 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60523f32-9f6f-4937-99c2-fc7aa9d96655-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx\" (UID: \"60523f32-9f6f-4937-99c2-fc7aa9d96655\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" Jul 10 16:46:47.491215 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.491201 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60523f32-9f6f-4937-99c2-fc7aa9d96655-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx\" (UID: \"60523f32-9f6f-4937-99c2-fc7aa9d96655\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" Jul 10 16:46:47.592412 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.592383 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60523f32-9f6f-4937-99c2-fc7aa9d96655-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx\" (UID: \"60523f32-9f6f-4937-99c2-fc7aa9d96655\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" Jul 10 16:46:47.592529 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.592453 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvfwp\" (UniqueName: \"kubernetes.io/projected/60523f32-9f6f-4937-99c2-fc7aa9d96655-kube-api-access-xvfwp\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx\" (UID: \"60523f32-9f6f-4937-99c2-fc7aa9d96655\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" Jul 10 16:46:47.592529 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.592495 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60523f32-9f6f-4937-99c2-fc7aa9d96655-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx\" (UID: \"60523f32-9f6f-4937-99c2-fc7aa9d96655\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" Jul 10 16:46:47.592856 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.592836 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60523f32-9f6f-4937-99c2-fc7aa9d96655-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx\" (UID: \"60523f32-9f6f-4937-99c2-fc7aa9d96655\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" Jul 10 16:46:47.592856 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.592849 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60523f32-9f6f-4937-99c2-fc7aa9d96655-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx\" (UID: \"60523f32-9f6f-4937-99c2-fc7aa9d96655\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" Jul 10 16:46:47.600913 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.600886 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvfwp\" (UniqueName: \"kubernetes.io/projected/60523f32-9f6f-4937-99c2-fc7aa9d96655-kube-api-access-xvfwp\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx\" (UID: \"60523f32-9f6f-4937-99c2-fc7aa9d96655\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" Jul 10 16:46:47.621349 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.621323 2585 generic.go:356] "Generic (PLEG): container finished" podID="b235db81-1b87-4a7c-8a5a-547ce8ca845a" containerID="fcd4024c2587cca1a9f3f0fa0304c9458b3f21f165ad26b43dcddb5e74a5dcc0" exitCode=0 Jul 10 16:46:47.621437 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.621356 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45" event={"ID":"b235db81-1b87-4a7c-8a5a-547ce8ca845a","Type":"ContainerDied","Data":"fcd4024c2587cca1a9f3f0fa0304c9458b3f21f165ad26b43dcddb5e74a5dcc0"} Jul 10 16:46:47.622933 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.622914 2585 generic.go:356] "Generic (PLEG): container finished" podID="c984eaa6-2d5a-41e1-8b6c-1def3e97c81a" containerID="b96074ecc050b191981942d3a407d9512d872311ce59e4d2425ae765164143b2" exitCode=0 Jul 10 16:46:47.623000 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.622946 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv" event={"ID":"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a","Type":"ContainerDied","Data":"b96074ecc050b191981942d3a407d9512d872311ce59e4d2425ae765164143b2"} Jul 10 16:46:47.623000 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.622978 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv" event={"ID":"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a","Type":"ContainerStarted","Data":"1349f3fa9b7f5fd89271d844f2fa4567b13ebe3e288017e80d23b2e6d3d91e0d"} Jul 10 16:46:47.657168 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.657139 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" Jul 10 16:46:47.737415 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.737370 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h"] Jul 10 16:46:47.741393 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.741372 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h" Jul 10 16:46:47.756376 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.756350 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h"] Jul 10 16:46:47.793018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.792924 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx"] Jul 10 16:46:47.808537 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:46:47.808512 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60523f32_9f6f_4937_99c2_fc7aa9d96655.slice/crio-ed684ce7acdcbcd6e75ccc71df6ee726cc0d9b86013a0ba85814d82df2373684 WatchSource:0}: Error finding container ed684ce7acdcbcd6e75ccc71df6ee726cc0d9b86013a0ba85814d82df2373684: Status 404 returned error can't find the container with id ed684ce7acdcbcd6e75ccc71df6ee726cc0d9b86013a0ba85814d82df2373684 Jul 10 16:46:47.894945 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.894921 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f997c02-2eaa-4bc0-8a27-e391e6530f64-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h\" (UID: \"1f997c02-2eaa-4bc0-8a27-e391e6530f64\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h" Jul 10 16:46:47.895050 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.895039 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcz79\" (UniqueName: \"kubernetes.io/projected/1f997c02-2eaa-4bc0-8a27-e391e6530f64-kube-api-access-vcz79\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h\" (UID: \"1f997c02-2eaa-4bc0-8a27-e391e6530f64\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h" Jul 10 16:46:47.895108 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.895065 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f997c02-2eaa-4bc0-8a27-e391e6530f64-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h\" (UID: \"1f997c02-2eaa-4bc0-8a27-e391e6530f64\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h" Jul 10 16:46:47.995988 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.995938 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f997c02-2eaa-4bc0-8a27-e391e6530f64-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h\" (UID: \"1f997c02-2eaa-4bc0-8a27-e391e6530f64\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h" Jul 10 16:46:47.996116 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.996019 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcz79\" (UniqueName: \"kubernetes.io/projected/1f997c02-2eaa-4bc0-8a27-e391e6530f64-kube-api-access-vcz79\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h\" (UID: \"1f997c02-2eaa-4bc0-8a27-e391e6530f64\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h" Jul 10 16:46:47.996116 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.996044 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f997c02-2eaa-4bc0-8a27-e391e6530f64-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h\" (UID: \"1f997c02-2eaa-4bc0-8a27-e391e6530f64\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h" Jul 10 16:46:47.996372 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.996351 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f997c02-2eaa-4bc0-8a27-e391e6530f64-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h\" (UID: \"1f997c02-2eaa-4bc0-8a27-e391e6530f64\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h" Jul 10 16:46:47.996413 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:47.996370 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f997c02-2eaa-4bc0-8a27-e391e6530f64-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h\" (UID: \"1f997c02-2eaa-4bc0-8a27-e391e6530f64\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h" Jul 10 16:46:48.004551 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:48.004523 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcz79\" (UniqueName: \"kubernetes.io/projected/1f997c02-2eaa-4bc0-8a27-e391e6530f64-kube-api-access-vcz79\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h\" (UID: \"1f997c02-2eaa-4bc0-8a27-e391e6530f64\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h" Jul 10 16:46:48.055124 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:48.055102 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h" Jul 10 16:46:48.180440 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:48.180417 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h"] Jul 10 16:46:48.182455 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:46:48.182428 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f997c02_2eaa_4bc0_8a27_e391e6530f64.slice/crio-45c35fe35839f5b009511c429582a3f3c87e3aade0818c66ff7e95c130a35adf WatchSource:0}: Error finding container 45c35fe35839f5b009511c429582a3f3c87e3aade0818c66ff7e95c130a35adf: Status 404 returned error can't find the container with id 45c35fe35839f5b009511c429582a3f3c87e3aade0818c66ff7e95c130a35adf Jul 10 16:46:48.635651 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:48.635623 2585 generic.go:356] "Generic (PLEG): container finished" podID="b235db81-1b87-4a7c-8a5a-547ce8ca845a" containerID="bc952c7579345807786d493222833efbb9feeb2ae08700d8be13358b09f109d3" exitCode=0 Jul 10 16:46:48.635738 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:48.635697 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45" event={"ID":"b235db81-1b87-4a7c-8a5a-547ce8ca845a","Type":"ContainerDied","Data":"bc952c7579345807786d493222833efbb9feeb2ae08700d8be13358b09f109d3"} Jul 10 16:46:48.637123 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:48.637101 2585 generic.go:356] "Generic (PLEG): container finished" podID="1f997c02-2eaa-4bc0-8a27-e391e6530f64" containerID="665d112639b5713e7e222143684c5ef3c1369a14ad0b40567029e27b35bf9e11" exitCode=0 Jul 10 16:46:48.637216 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:48.637171 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h" event={"ID":"1f997c02-2eaa-4bc0-8a27-e391e6530f64","Type":"ContainerDied","Data":"665d112639b5713e7e222143684c5ef3c1369a14ad0b40567029e27b35bf9e11"} Jul 10 16:46:48.637216 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:48.637208 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h" event={"ID":"1f997c02-2eaa-4bc0-8a27-e391e6530f64","Type":"ContainerStarted","Data":"45c35fe35839f5b009511c429582a3f3c87e3aade0818c66ff7e95c130a35adf"} Jul 10 16:46:48.638650 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:48.638632 2585 generic.go:356] "Generic (PLEG): container finished" podID="60523f32-9f6f-4937-99c2-fc7aa9d96655" containerID="3ab5d40617865a0d78c2e46ccdcd3a52e5a82468a795890e5678c8cd8a71136c" exitCode=0 Jul 10 16:46:48.638732 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:48.638676 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" event={"ID":"60523f32-9f6f-4937-99c2-fc7aa9d96655","Type":"ContainerDied","Data":"3ab5d40617865a0d78c2e46ccdcd3a52e5a82468a795890e5678c8cd8a71136c"} Jul 10 16:46:48.638732 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:48.638707 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" event={"ID":"60523f32-9f6f-4937-99c2-fc7aa9d96655","Type":"ContainerStarted","Data":"ed684ce7acdcbcd6e75ccc71df6ee726cc0d9b86013a0ba85814d82df2373684"} Jul 10 16:46:49.645868 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:49.645840 2585 generic.go:356] "Generic (PLEG): container finished" podID="c984eaa6-2d5a-41e1-8b6c-1def3e97c81a" containerID="cc129c0853809248f04df0c2abb3079fd0247ca82b9df5df09b133d4e175ffbd" exitCode=0 Jul 10 16:46:49.646183 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:49.645979 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv" event={"ID":"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a","Type":"ContainerDied","Data":"cc129c0853809248f04df0c2abb3079fd0247ca82b9df5df09b133d4e175ffbd"} Jul 10 16:46:49.647939 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:49.647913 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" event={"ID":"60523f32-9f6f-4937-99c2-fc7aa9d96655","Type":"ContainerStarted","Data":"15390f6d1668e922c54e2827bbd5ae7abc06a6aeca9b865a6d78a9de80495444"} Jul 10 16:46:49.918421 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:49.918397 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45" Jul 10 16:46:50.012542 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.012487 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b235db81-1b87-4a7c-8a5a-547ce8ca845a-bundle\") pod \"b235db81-1b87-4a7c-8a5a-547ce8ca845a\" (UID: \"b235db81-1b87-4a7c-8a5a-547ce8ca845a\") " Jul 10 16:46:50.012542 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.012540 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b235db81-1b87-4a7c-8a5a-547ce8ca845a-util\") pod \"b235db81-1b87-4a7c-8a5a-547ce8ca845a\" (UID: \"b235db81-1b87-4a7c-8a5a-547ce8ca845a\") " Jul 10 16:46:50.012674 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.012577 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bffp2\" (UniqueName: \"kubernetes.io/projected/b235db81-1b87-4a7c-8a5a-547ce8ca845a-kube-api-access-bffp2\") pod \"b235db81-1b87-4a7c-8a5a-547ce8ca845a\" (UID: \"b235db81-1b87-4a7c-8a5a-547ce8ca845a\") " Jul 10 16:46:50.013011 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.012979 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b235db81-1b87-4a7c-8a5a-547ce8ca845a-bundle" (OuterVolumeSpecName: "bundle") pod "b235db81-1b87-4a7c-8a5a-547ce8ca845a" (UID: "b235db81-1b87-4a7c-8a5a-547ce8ca845a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:46:50.014580 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.014554 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b235db81-1b87-4a7c-8a5a-547ce8ca845a-kube-api-access-bffp2" (OuterVolumeSpecName: "kube-api-access-bffp2") pod "b235db81-1b87-4a7c-8a5a-547ce8ca845a" (UID: "b235db81-1b87-4a7c-8a5a-547ce8ca845a"). InnerVolumeSpecName "kube-api-access-bffp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:46:50.017350 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.017330 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b235db81-1b87-4a7c-8a5a-547ce8ca845a-util" (OuterVolumeSpecName: "util") pod "b235db81-1b87-4a7c-8a5a-547ce8ca845a" (UID: "b235db81-1b87-4a7c-8a5a-547ce8ca845a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:46:50.113644 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.113615 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b235db81-1b87-4a7c-8a5a-547ce8ca845a-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:46:50.113644 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.113642 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b235db81-1b87-4a7c-8a5a-547ce8ca845a-util\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:46:50.113644 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.113652 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bffp2\" (UniqueName: \"kubernetes.io/projected/b235db81-1b87-4a7c-8a5a-547ce8ca845a-kube-api-access-bffp2\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:46:50.654970 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.654945 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45" Jul 10 16:46:50.654970 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.654963 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45" event={"ID":"b235db81-1b87-4a7c-8a5a-547ce8ca845a","Type":"ContainerDied","Data":"9e32520b85c0b82f0505fe5d73df0e0acd23617846031d9ecb8ab04b88769d7a"} Jul 10 16:46:50.655459 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.655011 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e32520b85c0b82f0505fe5d73df0e0acd23617846031d9ecb8ab04b88769d7a" Jul 10 16:46:50.657378 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.657356 2585 generic.go:356] "Generic (PLEG): container finished" podID="c984eaa6-2d5a-41e1-8b6c-1def3e97c81a" containerID="b7f0bc476e1786f3e4405c55690befe68e2f3909cb2faa9311c4129eacccf546" exitCode=0 Jul 10 16:46:50.657491 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.657441 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv" event={"ID":"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a","Type":"ContainerDied","Data":"b7f0bc476e1786f3e4405c55690befe68e2f3909cb2faa9311c4129eacccf546"} Jul 10 16:46:50.659119 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.659072 2585 generic.go:356] "Generic (PLEG): container finished" podID="1f997c02-2eaa-4bc0-8a27-e391e6530f64" containerID="e955732891deb8d1612aa1421895988468a4c7a632c3b5f40a085fd5b3829249" exitCode=0 Jul 10 16:46:50.659203 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.659143 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h" event={"ID":"1f997c02-2eaa-4bc0-8a27-e391e6530f64","Type":"ContainerDied","Data":"e955732891deb8d1612aa1421895988468a4c7a632c3b5f40a085fd5b3829249"} Jul 10 16:46:50.661836 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.661812 2585 generic.go:356] "Generic (PLEG): container finished" podID="60523f32-9f6f-4937-99c2-fc7aa9d96655" containerID="15390f6d1668e922c54e2827bbd5ae7abc06a6aeca9b865a6d78a9de80495444" exitCode=0 Jul 10 16:46:50.661836 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.661834 2585 generic.go:356] "Generic (PLEG): container finished" podID="60523f32-9f6f-4937-99c2-fc7aa9d96655" containerID="cd321fa93a0c5640f649c17479d87f5a672551dc7d1ee4694206530025594728" exitCode=0 Jul 10 16:46:50.661936 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.661887 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" event={"ID":"60523f32-9f6f-4937-99c2-fc7aa9d96655","Type":"ContainerDied","Data":"15390f6d1668e922c54e2827bbd5ae7abc06a6aeca9b865a6d78a9de80495444"} Jul 10 16:46:50.661936 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:50.661918 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" event={"ID":"60523f32-9f6f-4937-99c2-fc7aa9d96655","Type":"ContainerDied","Data":"cd321fa93a0c5640f649c17479d87f5a672551dc7d1ee4694206530025594728"} Jul 10 16:46:51.668685 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:51.668652 2585 generic.go:356] "Generic (PLEG): container finished" podID="1f997c02-2eaa-4bc0-8a27-e391e6530f64" containerID="5775102e029388a9446a79558cbb971484c8dc896256bd2757fc53255cd2d846" exitCode=0 Jul 10 16:46:51.669183 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:51.668735 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h" event={"ID":"1f997c02-2eaa-4bc0-8a27-e391e6530f64","Type":"ContainerDied","Data":"5775102e029388a9446a79558cbb971484c8dc896256bd2757fc53255cd2d846"} Jul 10 16:46:51.824139 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:51.824113 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv" Jul 10 16:46:51.826876 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:51.826859 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" Jul 10 16:46:51.931933 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:51.931876 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60523f32-9f6f-4937-99c2-fc7aa9d96655-bundle\") pod \"60523f32-9f6f-4937-99c2-fc7aa9d96655\" (UID: \"60523f32-9f6f-4937-99c2-fc7aa9d96655\") " Jul 10 16:46:51.931933 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:51.931911 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c984eaa6-2d5a-41e1-8b6c-1def3e97c81a-util\") pod \"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a\" (UID: \"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a\") " Jul 10 16:46:51.932128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:51.931948 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqspr\" (UniqueName: \"kubernetes.io/projected/c984eaa6-2d5a-41e1-8b6c-1def3e97c81a-kube-api-access-lqspr\") pod \"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a\" (UID: \"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a\") " Jul 10 16:46:51.932128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:51.932019 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c984eaa6-2d5a-41e1-8b6c-1def3e97c81a-bundle\") pod \"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a\" (UID: \"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a\") " Jul 10 16:46:51.932128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:51.932075 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvfwp\" (UniqueName: \"kubernetes.io/projected/60523f32-9f6f-4937-99c2-fc7aa9d96655-kube-api-access-xvfwp\") pod \"60523f32-9f6f-4937-99c2-fc7aa9d96655\" (UID: \"60523f32-9f6f-4937-99c2-fc7aa9d96655\") " Jul 10 16:46:51.932282 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:51.932140 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60523f32-9f6f-4937-99c2-fc7aa9d96655-util\") pod \"60523f32-9f6f-4937-99c2-fc7aa9d96655\" (UID: \"60523f32-9f6f-4937-99c2-fc7aa9d96655\") " Jul 10 16:46:51.932416 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:51.932393 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60523f32-9f6f-4937-99c2-fc7aa9d96655-bundle" (OuterVolumeSpecName: "bundle") pod "60523f32-9f6f-4937-99c2-fc7aa9d96655" (UID: "60523f32-9f6f-4937-99c2-fc7aa9d96655"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:46:51.932664 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:51.932616 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c984eaa6-2d5a-41e1-8b6c-1def3e97c81a-bundle" (OuterVolumeSpecName: "bundle") pod "c984eaa6-2d5a-41e1-8b6c-1def3e97c81a" (UID: "c984eaa6-2d5a-41e1-8b6c-1def3e97c81a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:46:51.934261 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:51.934233 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60523f32-9f6f-4937-99c2-fc7aa9d96655-kube-api-access-xvfwp" (OuterVolumeSpecName: "kube-api-access-xvfwp") pod "60523f32-9f6f-4937-99c2-fc7aa9d96655" (UID: "60523f32-9f6f-4937-99c2-fc7aa9d96655"). InnerVolumeSpecName "kube-api-access-xvfwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:46:51.934489 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:51.934470 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c984eaa6-2d5a-41e1-8b6c-1def3e97c81a-kube-api-access-lqspr" (OuterVolumeSpecName: "kube-api-access-lqspr") pod "c984eaa6-2d5a-41e1-8b6c-1def3e97c81a" (UID: "c984eaa6-2d5a-41e1-8b6c-1def3e97c81a"). InnerVolumeSpecName "kube-api-access-lqspr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:46:51.938322 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:51.938288 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60523f32-9f6f-4937-99c2-fc7aa9d96655-util" (OuterVolumeSpecName: "util") pod "60523f32-9f6f-4937-99c2-fc7aa9d96655" (UID: "60523f32-9f6f-4937-99c2-fc7aa9d96655"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:46:51.938429 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:51.938333 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c984eaa6-2d5a-41e1-8b6c-1def3e97c81a-util" (OuterVolumeSpecName: "util") pod "c984eaa6-2d5a-41e1-8b6c-1def3e97c81a" (UID: "c984eaa6-2d5a-41e1-8b6c-1def3e97c81a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:46:52.032991 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.032968 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c984eaa6-2d5a-41e1-8b6c-1def3e97c81a-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:46:52.032991 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.032990 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvfwp\" (UniqueName: \"kubernetes.io/projected/60523f32-9f6f-4937-99c2-fc7aa9d96655-kube-api-access-xvfwp\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:46:52.033108 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.033002 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60523f32-9f6f-4937-99c2-fc7aa9d96655-util\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:46:52.033108 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.033012 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60523f32-9f6f-4937-99c2-fc7aa9d96655-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:46:52.033108 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.033019 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c984eaa6-2d5a-41e1-8b6c-1def3e97c81a-util\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:46:52.033108 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.033027 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqspr\" (UniqueName: \"kubernetes.io/projected/c984eaa6-2d5a-41e1-8b6c-1def3e97c81a-kube-api-access-lqspr\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:46:52.674696 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.674667 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv" Jul 10 16:46:52.675192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.674699 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv" event={"ID":"c984eaa6-2d5a-41e1-8b6c-1def3e97c81a","Type":"ContainerDied","Data":"1349f3fa9b7f5fd89271d844f2fa4567b13ebe3e288017e80d23b2e6d3d91e0d"} Jul 10 16:46:52.675192 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.674766 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1349f3fa9b7f5fd89271d844f2fa4567b13ebe3e288017e80d23b2e6d3d91e0d" Jul 10 16:46:52.676540 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.676512 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" event={"ID":"60523f32-9f6f-4937-99c2-fc7aa9d96655","Type":"ContainerDied","Data":"ed684ce7acdcbcd6e75ccc71df6ee726cc0d9b86013a0ba85814d82df2373684"} Jul 10 16:46:52.676667 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.676562 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed684ce7acdcbcd6e75ccc71df6ee726cc0d9b86013a0ba85814d82df2373684" Jul 10 16:46:52.676667 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.676531 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx" Jul 10 16:46:52.793409 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.793387 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h" Jul 10 16:46:52.940806 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.940722 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f997c02-2eaa-4bc0-8a27-e391e6530f64-util\") pod \"1f997c02-2eaa-4bc0-8a27-e391e6530f64\" (UID: \"1f997c02-2eaa-4bc0-8a27-e391e6530f64\") " Jul 10 16:46:52.940942 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.940825 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f997c02-2eaa-4bc0-8a27-e391e6530f64-bundle\") pod \"1f997c02-2eaa-4bc0-8a27-e391e6530f64\" (UID: \"1f997c02-2eaa-4bc0-8a27-e391e6530f64\") " Jul 10 16:46:52.940942 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.940894 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcz79\" (UniqueName: \"kubernetes.io/projected/1f997c02-2eaa-4bc0-8a27-e391e6530f64-kube-api-access-vcz79\") pod \"1f997c02-2eaa-4bc0-8a27-e391e6530f64\" (UID: \"1f997c02-2eaa-4bc0-8a27-e391e6530f64\") " Jul 10 16:46:52.941307 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.941288 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f997c02-2eaa-4bc0-8a27-e391e6530f64-bundle" (OuterVolumeSpecName: "bundle") pod "1f997c02-2eaa-4bc0-8a27-e391e6530f64" (UID: "1f997c02-2eaa-4bc0-8a27-e391e6530f64"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:46:52.942963 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.942940 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f997c02-2eaa-4bc0-8a27-e391e6530f64-kube-api-access-vcz79" (OuterVolumeSpecName: "kube-api-access-vcz79") pod "1f997c02-2eaa-4bc0-8a27-e391e6530f64" (UID: "1f997c02-2eaa-4bc0-8a27-e391e6530f64"). InnerVolumeSpecName "kube-api-access-vcz79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:46:52.946353 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:52.946321 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f997c02-2eaa-4bc0-8a27-e391e6530f64-util" (OuterVolumeSpecName: "util") pod "1f997c02-2eaa-4bc0-8a27-e391e6530f64" (UID: "1f997c02-2eaa-4bc0-8a27-e391e6530f64"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:46:53.041872 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:53.041846 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f997c02-2eaa-4bc0-8a27-e391e6530f64-util\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:46:53.041872 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:53.041868 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f997c02-2eaa-4bc0-8a27-e391e6530f64-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:46:53.042002 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:53.041876 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcz79\" (UniqueName: \"kubernetes.io/projected/1f997c02-2eaa-4bc0-8a27-e391e6530f64-kube-api-access-vcz79\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:46:53.682779 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:53.682734 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h" event={"ID":"1f997c02-2eaa-4bc0-8a27-e391e6530f64","Type":"ContainerDied","Data":"45c35fe35839f5b009511c429582a3f3c87e3aade0818c66ff7e95c130a35adf"} Jul 10 16:46:53.682779 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:53.682773 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h" Jul 10 16:46:53.683133 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:46:53.682795 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45c35fe35839f5b009511c429582a3f3c87e3aade0818c66ff7e95c130a35adf" Jul 10 16:47:09.297146 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297070 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5l6zf"] Jul 10 16:47:09.297612 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297493 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b235db81-1b87-4a7c-8a5a-547ce8ca845a" containerName="extract" Jul 10 16:47:09.297612 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297513 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b235db81-1b87-4a7c-8a5a-547ce8ca845a" containerName="extract" Jul 10 16:47:09.297612 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297528 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60523f32-9f6f-4937-99c2-fc7aa9d96655" containerName="extract" Jul 10 16:47:09.297612 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297538 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="60523f32-9f6f-4937-99c2-fc7aa9d96655" containerName="extract" Jul 10 16:47:09.297612 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297547 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f997c02-2eaa-4bc0-8a27-e391e6530f64" containerName="extract" Jul 10 16:47:09.297612 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297553 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f997c02-2eaa-4bc0-8a27-e391e6530f64" containerName="extract" Jul 10 16:47:09.297612 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297560 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60523f32-9f6f-4937-99c2-fc7aa9d96655" containerName="pull" Jul 10 16:47:09.297612 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297566 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="60523f32-9f6f-4937-99c2-fc7aa9d96655" containerName="pull" Jul 10 16:47:09.297612 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297580 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60523f32-9f6f-4937-99c2-fc7aa9d96655" containerName="util" Jul 10 16:47:09.297612 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297588 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="60523f32-9f6f-4937-99c2-fc7aa9d96655" containerName="util" Jul 10 16:47:09.297612 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297597 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c984eaa6-2d5a-41e1-8b6c-1def3e97c81a" containerName="util" Jul 10 16:47:09.297612 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297604 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c984eaa6-2d5a-41e1-8b6c-1def3e97c81a" containerName="util" Jul 10 16:47:09.297612 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297616 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f997c02-2eaa-4bc0-8a27-e391e6530f64" containerName="util" Jul 10 16:47:09.298057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297624 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f997c02-2eaa-4bc0-8a27-e391e6530f64" containerName="util" Jul 10 16:47:09.298057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297632 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c984eaa6-2d5a-41e1-8b6c-1def3e97c81a" containerName="extract" Jul 10 16:47:09.298057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297638 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c984eaa6-2d5a-41e1-8b6c-1def3e97c81a" containerName="extract" Jul 10 16:47:09.298057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297646 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c984eaa6-2d5a-41e1-8b6c-1def3e97c81a" containerName="pull" Jul 10 16:47:09.298057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297651 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c984eaa6-2d5a-41e1-8b6c-1def3e97c81a" containerName="pull" Jul 10 16:47:09.298057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297658 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b235db81-1b87-4a7c-8a5a-547ce8ca845a" containerName="util" Jul 10 16:47:09.298057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297663 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b235db81-1b87-4a7c-8a5a-547ce8ca845a" containerName="util" Jul 10 16:47:09.298057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297669 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f997c02-2eaa-4bc0-8a27-e391e6530f64" containerName="pull" Jul 10 16:47:09.298057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297673 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f997c02-2eaa-4bc0-8a27-e391e6530f64" containerName="pull" Jul 10 16:47:09.298057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297682 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b235db81-1b87-4a7c-8a5a-547ce8ca845a" containerName="pull" Jul 10 16:47:09.298057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297691 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b235db81-1b87-4a7c-8a5a-547ce8ca845a" containerName="pull" Jul 10 16:47:09.298057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297796 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f997c02-2eaa-4bc0-8a27-e391e6530f64" containerName="extract" Jul 10 16:47:09.298057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297810 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c984eaa6-2d5a-41e1-8b6c-1def3e97c81a" containerName="extract" Jul 10 16:47:09.298057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297816 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b235db81-1b87-4a7c-8a5a-547ce8ca845a" containerName="extract" Jul 10 16:47:09.298057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.297823 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="60523f32-9f6f-4937-99c2-fc7aa9d96655" containerName="extract" Jul 10 16:47:09.305112 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.305094 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5l6zf" Jul 10 16:47:09.308858 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.308836 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 16:47:09.308963 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.308871 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7fnqz\"" Jul 10 16:47:09.315363 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.315341 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5l6zf"] Jul 10 16:47:09.462277 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.462254 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc8cf\" (UniqueName: \"kubernetes.io/projected/f23fb5ac-7e1f-4fd4-99bc-b368b3204ff0-kube-api-access-dc8cf\") pod \"dns-operator-controller-manager-864dd78c4c-5l6zf\" (UID: \"f23fb5ac-7e1f-4fd4-99bc-b368b3204ff0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5l6zf" Jul 10 16:47:09.563133 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.563072 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc8cf\" (UniqueName: \"kubernetes.io/projected/f23fb5ac-7e1f-4fd4-99bc-b368b3204ff0-kube-api-access-dc8cf\") pod \"dns-operator-controller-manager-864dd78c4c-5l6zf\" (UID: \"f23fb5ac-7e1f-4fd4-99bc-b368b3204ff0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5l6zf" Jul 10 16:47:09.572692 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.572671 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc8cf\" (UniqueName: \"kubernetes.io/projected/f23fb5ac-7e1f-4fd4-99bc-b368b3204ff0-kube-api-access-dc8cf\") pod \"dns-operator-controller-manager-864dd78c4c-5l6zf\" (UID: \"f23fb5ac-7e1f-4fd4-99bc-b368b3204ff0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5l6zf" Jul 10 16:47:09.615637 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.615617 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5l6zf" Jul 10 16:47:09.747051 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.747024 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5l6zf"] Jul 10 16:47:09.748980 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:47:09.748955 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23fb5ac_7e1f_4fd4_99bc_b368b3204ff0.slice/crio-77a62a0fb575f20ecba8106f3d547bc73dc57c234c0e17522004241b008baf78 WatchSource:0}: Error finding container 77a62a0fb575f20ecba8106f3d547bc73dc57c234c0e17522004241b008baf78: Status 404 returned error can't find the container with id 77a62a0fb575f20ecba8106f3d547bc73dc57c234c0e17522004241b008baf78 Jul 10 16:47:09.758813 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:09.758788 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5l6zf" event={"ID":"f23fb5ac-7e1f-4fd4-99bc-b368b3204ff0","Type":"ContainerStarted","Data":"77a62a0fb575f20ecba8106f3d547bc73dc57c234c0e17522004241b008baf78"} Jul 10 16:47:12.776648 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:12.776604 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5l6zf" event={"ID":"f23fb5ac-7e1f-4fd4-99bc-b368b3204ff0","Type":"ContainerStarted","Data":"49754a09cbe45a05c8644720aa8b31df0c3379375a267fcf611f80bc0459631f"} Jul 10 16:47:12.777031 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:12.776676 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5l6zf" Jul 10 16:47:12.788757 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:12.788724 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj"] Jul 10 16:47:12.792223 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:12.792203 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" Jul 10 16:47:12.795178 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:12.795160 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-ntrmm\"" Jul 10 16:47:12.798507 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:12.798462 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5l6zf" podStartSLOduration=1.571120072 podStartE2EDuration="3.7984451s" totalImagesPullingTime="2.227325028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:47:12.797635403 +0000 UTC m=+652.219511904" watchObservedRunningTime="2026-07-10 16:47:12.7984451 +0000 UTC m=+652.220321563" Jul 10 16:47:12.805057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:12.805034 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj"] Jul 10 16:47:12.892116 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:12.892033 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j59cc\" (UniqueName: \"kubernetes.io/projected/fb4daf3d-5927-44b3-9452-85e4c6f6c94b-kube-api-access-j59cc\") pod \"limitador-operator-controller-manager-859d7b78bd-wrdwj\" (UID: \"fb4daf3d-5927-44b3-9452-85e4c6f6c94b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" Jul 10 16:47:12.992965 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:12.992922 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j59cc\" (UniqueName: \"kubernetes.io/projected/fb4daf3d-5927-44b3-9452-85e4c6f6c94b-kube-api-access-j59cc\") pod \"limitador-operator-controller-manager-859d7b78bd-wrdwj\" (UID: \"fb4daf3d-5927-44b3-9452-85e4c6f6c94b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" Jul 10 16:47:13.001514 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:13.001489 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j59cc\" (UniqueName: \"kubernetes.io/projected/fb4daf3d-5927-44b3-9452-85e4c6f6c94b-kube-api-access-j59cc\") pod \"limitador-operator-controller-manager-859d7b78bd-wrdwj\" (UID: \"fb4daf3d-5927-44b3-9452-85e4c6f6c94b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" Jul 10 16:47:13.103843 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:13.103755 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" Jul 10 16:47:13.234627 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:13.234606 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj"] Jul 10 16:47:13.236929 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:47:13.236904 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb4daf3d_5927_44b3_9452_85e4c6f6c94b.slice/crio-c9eaa43e5781af3acee64a9b3f4eef804c3369f0e79214e5fd38e990a779deb5 WatchSource:0}: Error finding container c9eaa43e5781af3acee64a9b3f4eef804c3369f0e79214e5fd38e990a779deb5: Status 404 returned error can't find the container with id c9eaa43e5781af3acee64a9b3f4eef804c3369f0e79214e5fd38e990a779deb5 Jul 10 16:47:13.785715 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:13.785671 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" event={"ID":"fb4daf3d-5927-44b3-9452-85e4c6f6c94b","Type":"ContainerStarted","Data":"c9eaa43e5781af3acee64a9b3f4eef804c3369f0e79214e5fd38e990a779deb5"} Jul 10 16:47:15.798220 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:15.798184 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" event={"ID":"fb4daf3d-5927-44b3-9452-85e4c6f6c94b","Type":"ContainerStarted","Data":"ac418c32e65b68eb8420e5a2e1a7d36b7bc8b0db0c1809e07027ba4361d68052"} Jul 10 16:47:15.798645 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:15.798246 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" Jul 10 16:47:15.818775 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:15.818714 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" podStartSLOduration=2.113291323 podStartE2EDuration="3.818701038s" totalImagesPullingTime="1.705409715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:47:15.816210245 +0000 UTC m=+655.238086712" watchObservedRunningTime="2026-07-10 16:47:15.818701038 +0000 UTC m=+655.240577505" Jul 10 16:47:17.783227 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:17.783192 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq"] Jul 10 16:47:17.786861 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:17.786842 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:17.789613 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:17.789598 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m8z57\"" Jul 10 16:47:17.800858 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:17.800838 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq"] Jul 10 16:47:17.836005 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:17.835960 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f07ee45-4540-4698-821a-c56c8a3382b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2ftlq\" (UID: \"3f07ee45-4540-4698-821a-c56c8a3382b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:17.836005 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:17.836006 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj64x\" (UniqueName: \"kubernetes.io/projected/3f07ee45-4540-4698-821a-c56c8a3382b8-kube-api-access-mj64x\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2ftlq\" (UID: \"3f07ee45-4540-4698-821a-c56c8a3382b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:17.936586 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:17.936548 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f07ee45-4540-4698-821a-c56c8a3382b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2ftlq\" (UID: \"3f07ee45-4540-4698-821a-c56c8a3382b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:17.936586 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:17.936587 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj64x\" (UniqueName: \"kubernetes.io/projected/3f07ee45-4540-4698-821a-c56c8a3382b8-kube-api-access-mj64x\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2ftlq\" (UID: \"3f07ee45-4540-4698-821a-c56c8a3382b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:17.936935 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:17.936915 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f07ee45-4540-4698-821a-c56c8a3382b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2ftlq\" (UID: \"3f07ee45-4540-4698-821a-c56c8a3382b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:17.950233 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:17.950205 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj64x\" (UniqueName: \"kubernetes.io/projected/3f07ee45-4540-4698-821a-c56c8a3382b8-kube-api-access-mj64x\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2ftlq\" (UID: \"3f07ee45-4540-4698-821a-c56c8a3382b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:18.096211 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:18.096145 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:18.253551 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:18.253517 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq"] Jul 10 16:47:18.256782 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:47:18.256756 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f07ee45_4540_4698_821a_c56c8a3382b8.slice/crio-a2e2f612d71468fb42b01836c1e17f6ec17c0c4cc2d497d9656f1ac083bb61e5 WatchSource:0}: Error finding container a2e2f612d71468fb42b01836c1e17f6ec17c0c4cc2d497d9656f1ac083bb61e5: Status 404 returned error can't find the container with id a2e2f612d71468fb42b01836c1e17f6ec17c0c4cc2d497d9656f1ac083bb61e5 Jul 10 16:47:18.814811 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:18.814777 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" event={"ID":"3f07ee45-4540-4698-821a-c56c8a3382b8","Type":"ContainerStarted","Data":"a2e2f612d71468fb42b01836c1e17f6ec17c0c4cc2d497d9656f1ac083bb61e5"} Jul 10 16:47:23.788407 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:23.788354 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5l6zf" Jul 10 16:47:24.850181 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:24.850149 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" event={"ID":"3f07ee45-4540-4698-821a-c56c8a3382b8","Type":"ContainerStarted","Data":"5d891c0e2e861c38dc6431bf70a1fbbc98314e336c123ca8989be6bccc1fecf8"} Jul 10 16:47:24.850700 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:24.850207 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:24.874659 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:24.874614 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" podStartSLOduration=2.083024781 podStartE2EDuration="7.874598656s" totalImagesPullingTime="5.791573875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:47:24.87260425 +0000 UTC m=+664.294480718" watchObservedRunningTime="2026-07-10 16:47:24.874598656 +0000 UTC m=+664.296475121" Jul 10 16:47:26.805376 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:26.805346 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" Jul 10 16:47:26.842915 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:26.842883 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-sgsw7"] Jul 10 16:47:26.846856 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:26.846832 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sgsw7" Jul 10 16:47:26.849577 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:26.849554 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 16:47:26.849685 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:26.849554 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pzmp4\"" Jul 10 16:47:26.850169 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:26.850151 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 16:47:26.857044 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:26.857021 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-sgsw7"] Jul 10 16:47:26.921790 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:26.921731 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1780a33-28d0-4f8c-9388-91eba6f0c996-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-sgsw7\" (UID: \"a1780a33-28d0-4f8c-9388-91eba6f0c996\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sgsw7" Jul 10 16:47:26.921947 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:26.921835 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdnhv\" (UniqueName: \"kubernetes.io/projected/a1780a33-28d0-4f8c-9388-91eba6f0c996-kube-api-access-mdnhv\") pod \"kuadrant-console-plugin-66b5774985-sgsw7\" (UID: \"a1780a33-28d0-4f8c-9388-91eba6f0c996\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sgsw7" Jul 10 16:47:26.921947 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:26.921886 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a1780a33-28d0-4f8c-9388-91eba6f0c996-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-sgsw7\" (UID: \"a1780a33-28d0-4f8c-9388-91eba6f0c996\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sgsw7" Jul 10 16:47:27.022995 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:27.022955 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1780a33-28d0-4f8c-9388-91eba6f0c996-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-sgsw7\" (UID: \"a1780a33-28d0-4f8c-9388-91eba6f0c996\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sgsw7" Jul 10 16:47:27.023160 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:27.023015 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdnhv\" (UniqueName: \"kubernetes.io/projected/a1780a33-28d0-4f8c-9388-91eba6f0c996-kube-api-access-mdnhv\") pod \"kuadrant-console-plugin-66b5774985-sgsw7\" (UID: \"a1780a33-28d0-4f8c-9388-91eba6f0c996\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sgsw7" Jul 10 16:47:27.023160 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:27.023065 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a1780a33-28d0-4f8c-9388-91eba6f0c996-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-sgsw7\" (UID: \"a1780a33-28d0-4f8c-9388-91eba6f0c996\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sgsw7" Jul 10 16:47:27.023160 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:27.023068 2585 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 10 16:47:27.023160 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:27.023133 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1780a33-28d0-4f8c-9388-91eba6f0c996-plugin-serving-cert podName:a1780a33-28d0-4f8c-9388-91eba6f0c996 nodeName:}" failed. No retries permitted until 2026-07-10 16:47:27.52311486 +0000 UTC m=+666.944991308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/a1780a33-28d0-4f8c-9388-91eba6f0c996-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-sgsw7" (UID: "a1780a33-28d0-4f8c-9388-91eba6f0c996") : secret "plugin-serving-cert" not found Jul 10 16:47:27.023689 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:27.023673 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a1780a33-28d0-4f8c-9388-91eba6f0c996-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-sgsw7\" (UID: \"a1780a33-28d0-4f8c-9388-91eba6f0c996\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sgsw7" Jul 10 16:47:27.044796 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:27.044757 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdnhv\" (UniqueName: \"kubernetes.io/projected/a1780a33-28d0-4f8c-9388-91eba6f0c996-kube-api-access-mdnhv\") pod \"kuadrant-console-plugin-66b5774985-sgsw7\" (UID: \"a1780a33-28d0-4f8c-9388-91eba6f0c996\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sgsw7" Jul 10 16:47:27.528208 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:27.528172 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1780a33-28d0-4f8c-9388-91eba6f0c996-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-sgsw7\" (UID: \"a1780a33-28d0-4f8c-9388-91eba6f0c996\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sgsw7" Jul 10 16:47:27.530695 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:27.530658 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1780a33-28d0-4f8c-9388-91eba6f0c996-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-sgsw7\" (UID: \"a1780a33-28d0-4f8c-9388-91eba6f0c996\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sgsw7" Jul 10 16:47:27.757324 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:27.757289 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sgsw7" Jul 10 16:47:27.888809 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:27.888787 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-sgsw7"] Jul 10 16:47:27.890560 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:47:27.890525 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1780a33_28d0_4f8c_9388_91eba6f0c996.slice/crio-b3bff0dcfb0b2260d91a34b8d9ea14092516bcf64fff3ff143fd072e946f92b8 WatchSource:0}: Error finding container b3bff0dcfb0b2260d91a34b8d9ea14092516bcf64fff3ff143fd072e946f92b8: Status 404 returned error can't find the container with id b3bff0dcfb0b2260d91a34b8d9ea14092516bcf64fff3ff143fd072e946f92b8 Jul 10 16:47:28.875732 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:28.875694 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sgsw7" event={"ID":"a1780a33-28d0-4f8c-9388-91eba6f0c996","Type":"ContainerStarted","Data":"b3bff0dcfb0b2260d91a34b8d9ea14092516bcf64fff3ff143fd072e946f92b8"} Jul 10 16:47:35.857983 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:35.857947 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:37.616950 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.616910 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq"] Jul 10 16:47:37.617383 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.617147 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" podUID="3f07ee45-4540-4698-821a-c56c8a3382b8" containerName="manager" containerID="cri-o://5d891c0e2e861c38dc6431bf70a1fbbc98314e336c123ca8989be6bccc1fecf8" gracePeriod=2 Jul 10 16:47:37.628729 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.628684 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq"] Jul 10 16:47:37.629018 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:37.628676 2585 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3f07ee45-4540-4698-821a-c56c8a3382b8\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-2ftlq\": pods \"kuadrant-operator-controller-manager-8684c9ff49-2ftlq\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:37.644648 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.644610 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2"] Jul 10 16:47:37.645167 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.645144 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f07ee45-4540-4698-821a-c56c8a3382b8" containerName="manager" Jul 10 16:47:37.645167 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.645169 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f07ee45-4540-4698-821a-c56c8a3382b8" containerName="manager" Jul 10 16:47:37.645365 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.645260 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f07ee45-4540-4698-821a-c56c8a3382b8" containerName="manager" Jul 10 16:47:37.650011 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.648772 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj"] Jul 10 16:47:37.650011 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.648993 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" podUID="fb4daf3d-5927-44b3-9452-85e4c6f6c94b" containerName="manager" containerID="cri-o://ac418c32e65b68eb8420e5a2e1a7d36b7bc8b0db0c1809e07027ba4361d68052" gracePeriod=2 Jul 10 16:47:37.650011 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.649229 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2" Jul 10 16:47:37.656687 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.656654 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj"] Jul 10 16:47:37.676136 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.675283 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2"] Jul 10 16:47:37.678186 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.678160 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4fkhd"] Jul 10 16:47:37.678530 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.678514 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4daf3d-5927-44b3-9452-85e4c6f6c94b" containerName="manager" Jul 10 16:47:37.678530 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.678529 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4daf3d-5927-44b3-9452-85e4c6f6c94b" containerName="manager" Jul 10 16:47:37.678661 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.678589 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb4daf3d-5927-44b3-9452-85e4c6f6c94b" containerName="manager" Jul 10 16:47:37.683035 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.683007 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4fkhd" Jul 10 16:47:37.686563 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.686532 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4fkhd"] Jul 10 16:47:37.696106 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:37.695954 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2ftlq\" is forbidden: User \"system:node:ip-10-0-134-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-41.ec2.internal' and this object" podUID="3f07ee45-4540-4698-821a-c56c8a3382b8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:37.698712 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:37.698676 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wrdwj\" is forbidden: User \"system:node:ip-10-0-134-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-41.ec2.internal' and this object" podUID="fb4daf3d-5927-44b3-9452-85e4c6f6c94b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" Jul 10 16:47:37.720509 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:37.720456 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2ftlq\" is forbidden: User \"system:node:ip-10-0-134-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-41.ec2.internal' and this object" podUID="3f07ee45-4540-4698-821a-c56c8a3382b8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:37.722583 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:37.722546 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wrdwj\" is forbidden: User \"system:node:ip-10-0-134-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-41.ec2.internal' and this object" podUID="fb4daf3d-5927-44b3-9452-85e4c6f6c94b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" Jul 10 16:47:37.727503 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.727466 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddc62214-6483-4a44-ae77-871cc8cb7d42-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bhwn2\" (UID: \"ddc62214-6483-4a44-ae77-871cc8cb7d42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2" Jul 10 16:47:37.727665 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.727526 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxw8h\" (UniqueName: \"kubernetes.io/projected/ddc62214-6483-4a44-ae77-871cc8cb7d42-kube-api-access-mxw8h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bhwn2\" (UID: \"ddc62214-6483-4a44-ae77-871cc8cb7d42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2" Jul 10 16:47:37.828403 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.828366 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7dvl\" (UniqueName: \"kubernetes.io/projected/52147cde-c5c4-4dcb-9a5b-f1ec2d81efa2-kube-api-access-c7dvl\") pod \"limitador-operator-controller-manager-859d7b78bd-4fkhd\" (UID: \"52147cde-c5c4-4dcb-9a5b-f1ec2d81efa2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4fkhd" Jul 10 16:47:37.828621 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.828413 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddc62214-6483-4a44-ae77-871cc8cb7d42-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bhwn2\" (UID: \"ddc62214-6483-4a44-ae77-871cc8cb7d42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2" Jul 10 16:47:37.828621 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.828461 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxw8h\" (UniqueName: \"kubernetes.io/projected/ddc62214-6483-4a44-ae77-871cc8cb7d42-kube-api-access-mxw8h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bhwn2\" (UID: \"ddc62214-6483-4a44-ae77-871cc8cb7d42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2" Jul 10 16:47:37.835073 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.829642 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddc62214-6483-4a44-ae77-871cc8cb7d42-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bhwn2\" (UID: \"ddc62214-6483-4a44-ae77-871cc8cb7d42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2" Jul 10 16:47:37.839974 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.839937 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxw8h\" (UniqueName: \"kubernetes.io/projected/ddc62214-6483-4a44-ae77-871cc8cb7d42-kube-api-access-mxw8h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bhwn2\" (UID: \"ddc62214-6483-4a44-ae77-871cc8cb7d42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2" Jul 10 16:47:37.929720 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.929623 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7dvl\" (UniqueName: \"kubernetes.io/projected/52147cde-c5c4-4dcb-9a5b-f1ec2d81efa2-kube-api-access-c7dvl\") pod \"limitador-operator-controller-manager-859d7b78bd-4fkhd\" (UID: \"52147cde-c5c4-4dcb-9a5b-f1ec2d81efa2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4fkhd" Jul 10 16:47:37.942237 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:37.942196 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7dvl\" (UniqueName: \"kubernetes.io/projected/52147cde-c5c4-4dcb-9a5b-f1ec2d81efa2-kube-api-access-c7dvl\") pod \"limitador-operator-controller-manager-859d7b78bd-4fkhd\" (UID: \"52147cde-c5c4-4dcb-9a5b-f1ec2d81efa2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4fkhd" Jul 10 16:47:38.036144 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:38.036092 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2" Jul 10 16:47:38.044212 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:38.044176 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4fkhd" Jul 10 16:47:50.383530 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:50.383508 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" Jul 10 16:47:50.385538 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:50.385517 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:50.386177 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:50.386150 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wrdwj\" is forbidden: User \"system:node:ip-10-0-134-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-41.ec2.internal' and this object" podUID="fb4daf3d-5927-44b3-9452-85e4c6f6c94b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" Jul 10 16:47:50.389505 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:50.389470 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2ftlq\" is forbidden: User \"system:node:ip-10-0-134-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-41.ec2.internal' and this object" podUID="3f07ee45-4540-4698-821a-c56c8a3382b8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:50.392220 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:50.392178 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wrdwj\" is forbidden: User \"system:node:ip-10-0-134-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-41.ec2.internal' and this object" podUID="fb4daf3d-5927-44b3-9452-85e4c6f6c94b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" Jul 10 16:47:50.392295 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:50.392224 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2"] Jul 10 16:47:50.393928 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:47:50.393908 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc62214_6483_4a44_ae77_871cc8cb7d42.slice/crio-ae8557377bc84891ba93643a570d8cb8f9c26a7a221ff54cbce68f7a91a23e25 WatchSource:0}: Error finding container ae8557377bc84891ba93643a570d8cb8f9c26a7a221ff54cbce68f7a91a23e25: Status 404 returned error can't find the container with id ae8557377bc84891ba93643a570d8cb8f9c26a7a221ff54cbce68f7a91a23e25 Jul 10 16:47:50.403940 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:50.403918 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4fkhd"] Jul 10 16:47:50.405456 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:47:50.405401 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52147cde_c5c4_4dcb_9a5b_f1ec2d81efa2.slice/crio-930d517b284950c1e6391c59b90d864ca7aa1a372b23b997e54570c44bd4198b WatchSource:0}: Error finding container 930d517b284950c1e6391c59b90d864ca7aa1a372b23b997e54570c44bd4198b: Status 404 returned error can't find the container with id 930d517b284950c1e6391c59b90d864ca7aa1a372b23b997e54570c44bd4198b Jul 10 16:47:50.555907 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:50.555874 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj64x\" (UniqueName: \"kubernetes.io/projected/3f07ee45-4540-4698-821a-c56c8a3382b8-kube-api-access-mj64x\") pod \"3f07ee45-4540-4698-821a-c56c8a3382b8\" (UID: \"3f07ee45-4540-4698-821a-c56c8a3382b8\") " Jul 10 16:47:50.556057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:50.555929 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f07ee45-4540-4698-821a-c56c8a3382b8-extensions-socket-volume\") pod \"3f07ee45-4540-4698-821a-c56c8a3382b8\" (UID: \"3f07ee45-4540-4698-821a-c56c8a3382b8\") " Jul 10 16:47:50.556057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:50.556028 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j59cc\" (UniqueName: \"kubernetes.io/projected/fb4daf3d-5927-44b3-9452-85e4c6f6c94b-kube-api-access-j59cc\") pod \"fb4daf3d-5927-44b3-9452-85e4c6f6c94b\" (UID: \"fb4daf3d-5927-44b3-9452-85e4c6f6c94b\") " Jul 10 16:47:50.556373 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:50.556349 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f07ee45-4540-4698-821a-c56c8a3382b8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3f07ee45-4540-4698-821a-c56c8a3382b8" (UID: "3f07ee45-4540-4698-821a-c56c8a3382b8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:47:50.557984 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:50.557962 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4daf3d-5927-44b3-9452-85e4c6f6c94b-kube-api-access-j59cc" (OuterVolumeSpecName: "kube-api-access-j59cc") pod "fb4daf3d-5927-44b3-9452-85e4c6f6c94b" (UID: "fb4daf3d-5927-44b3-9452-85e4c6f6c94b"). InnerVolumeSpecName "kube-api-access-j59cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:47:50.558091 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:50.558071 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f07ee45-4540-4698-821a-c56c8a3382b8-kube-api-access-mj64x" (OuterVolumeSpecName: "kube-api-access-mj64x") pod "3f07ee45-4540-4698-821a-c56c8a3382b8" (UID: "3f07ee45-4540-4698-821a-c56c8a3382b8"). InnerVolumeSpecName "kube-api-access-mj64x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:47:50.657632 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:50.657597 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j59cc\" (UniqueName: \"kubernetes.io/projected/fb4daf3d-5927-44b3-9452-85e4c6f6c94b-kube-api-access-j59cc\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:47:50.657632 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:50.657630 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj64x\" (UniqueName: \"kubernetes.io/projected/3f07ee45-4540-4698-821a-c56c8a3382b8-kube-api-access-mj64x\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:47:50.657632 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:50.657639 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f07ee45-4540-4698-821a-c56c8a3382b8-extensions-socket-volume\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:47:51.016462 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.016424 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4fkhd" event={"ID":"52147cde-c5c4-4dcb-9a5b-f1ec2d81efa2","Type":"ContainerStarted","Data":"7ec962ab09c0396d0d7cdc2f40415942222eef9b34b371be536ab03eb752bbd1"} Jul 10 16:47:51.016652 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.016493 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4fkhd" event={"ID":"52147cde-c5c4-4dcb-9a5b-f1ec2d81efa2","Type":"ContainerStarted","Data":"930d517b284950c1e6391c59b90d864ca7aa1a372b23b997e54570c44bd4198b"} Jul 10 16:47:51.016652 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.016528 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4fkhd" Jul 10 16:47:51.017699 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.017676 2585 generic.go:356] "Generic (PLEG): container finished" podID="3f07ee45-4540-4698-821a-c56c8a3382b8" containerID="5d891c0e2e861c38dc6431bf70a1fbbc98314e336c123ca8989be6bccc1fecf8" exitCode=0 Jul 10 16:47:51.017791 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.017708 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:51.017791 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.017763 2585 scope.go:117] "RemoveContainer" containerID="5d891c0e2e861c38dc6431bf70a1fbbc98314e336c123ca8989be6bccc1fecf8" Jul 10 16:47:51.019697 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.019667 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2" event={"ID":"ddc62214-6483-4a44-ae77-871cc8cb7d42","Type":"ContainerStarted","Data":"f42405332bf43b3b8bd819d10103cded21d79dc32853f9c60fadfbc9a25418da"} Jul 10 16:47:51.019851 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.019717 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2" event={"ID":"ddc62214-6483-4a44-ae77-871cc8cb7d42","Type":"ContainerStarted","Data":"ae8557377bc84891ba93643a570d8cb8f9c26a7a221ff54cbce68f7a91a23e25"} Jul 10 16:47:51.019851 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.019805 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2" Jul 10 16:47:51.021313 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.021292 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sgsw7" event={"ID":"a1780a33-28d0-4f8c-9388-91eba6f0c996","Type":"ContainerStarted","Data":"a5ae8ad074268ac2b4add9162fe115fd43369abfb6a6bc4abf269893e64cb178"} Jul 10 16:47:51.022551 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.022529 2585 generic.go:356] "Generic (PLEG): container finished" podID="fb4daf3d-5927-44b3-9452-85e4c6f6c94b" containerID="ac418c32e65b68eb8420e5a2e1a7d36b7bc8b0db0c1809e07027ba4361d68052" exitCode=0 Jul 10 16:47:51.022658 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.022561 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" Jul 10 16:47:51.027188 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.027173 2585 scope.go:117] "RemoveContainer" containerID="5d891c0e2e861c38dc6431bf70a1fbbc98314e336c123ca8989be6bccc1fecf8" Jul 10 16:47:51.027435 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:51.027417 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d891c0e2e861c38dc6431bf70a1fbbc98314e336c123ca8989be6bccc1fecf8\": container with ID starting with 5d891c0e2e861c38dc6431bf70a1fbbc98314e336c123ca8989be6bccc1fecf8 not found: ID does not exist" containerID="5d891c0e2e861c38dc6431bf70a1fbbc98314e336c123ca8989be6bccc1fecf8" Jul 10 16:47:51.027476 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.027443 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d891c0e2e861c38dc6431bf70a1fbbc98314e336c123ca8989be6bccc1fecf8"} err="failed to get container status \"5d891c0e2e861c38dc6431bf70a1fbbc98314e336c123ca8989be6bccc1fecf8\": rpc error: code = NotFound desc = could not find container \"5d891c0e2e861c38dc6431bf70a1fbbc98314e336c123ca8989be6bccc1fecf8\": container with ID starting with 5d891c0e2e861c38dc6431bf70a1fbbc98314e336c123ca8989be6bccc1fecf8 not found: ID does not exist" Jul 10 16:47:51.027476 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.027458 2585 scope.go:117] "RemoveContainer" containerID="ac418c32e65b68eb8420e5a2e1a7d36b7bc8b0db0c1809e07027ba4361d68052" Jul 10 16:47:51.034774 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.034709 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4fkhd" podStartSLOduration=14.034693881 podStartE2EDuration="14.034693881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:47:51.034068314 +0000 UTC m=+690.455944781" watchObservedRunningTime="2026-07-10 16:47:51.034693881 +0000 UTC m=+690.456570348" Jul 10 16:47:51.036359 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:51.036334 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2ftlq\" is forbidden: User \"system:node:ip-10-0-134-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-41.ec2.internal' and this object" podUID="3f07ee45-4540-4698-821a-c56c8a3382b8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:51.036705 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.036692 2585 scope.go:117] "RemoveContainer" containerID="ac418c32e65b68eb8420e5a2e1a7d36b7bc8b0db0c1809e07027ba4361d68052" Jul 10 16:47:51.037025 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:51.037003 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac418c32e65b68eb8420e5a2e1a7d36b7bc8b0db0c1809e07027ba4361d68052\": container with ID starting with ac418c32e65b68eb8420e5a2e1a7d36b7bc8b0db0c1809e07027ba4361d68052 not found: ID does not exist" containerID="ac418c32e65b68eb8420e5a2e1a7d36b7bc8b0db0c1809e07027ba4361d68052" Jul 10 16:47:51.037102 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.037034 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac418c32e65b68eb8420e5a2e1a7d36b7bc8b0db0c1809e07027ba4361d68052"} err="failed to get container status \"ac418c32e65b68eb8420e5a2e1a7d36b7bc8b0db0c1809e07027ba4361d68052\": rpc error: code = NotFound desc = could not find container \"ac418c32e65b68eb8420e5a2e1a7d36b7bc8b0db0c1809e07027ba4361d68052\": container with ID starting with ac418c32e65b68eb8420e5a2e1a7d36b7bc8b0db0c1809e07027ba4361d68052 not found: ID does not exist" Jul 10 16:47:51.038462 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:51.038438 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wrdwj\" is forbidden: User \"system:node:ip-10-0-134-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-41.ec2.internal' and this object" podUID="fb4daf3d-5927-44b3-9452-85e4c6f6c94b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" Jul 10 16:47:51.059331 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.059286 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2" podStartSLOduration=14.059268826 podStartE2EDuration="14.059268826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:47:51.059224076 +0000 UTC m=+690.481100541" watchObservedRunningTime="2026-07-10 16:47:51.059268826 +0000 UTC m=+690.481145297" Jul 10 16:47:51.061611 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:51.061586 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2ftlq\" is forbidden: User \"system:node:ip-10-0-134-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-41.ec2.internal' and this object" podUID="3f07ee45-4540-4698-821a-c56c8a3382b8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:51.078461 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.078415 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sgsw7" podStartSLOduration=2.663259699 podStartE2EDuration="25.078399482s" totalImagesPullingTime="22.415139783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:47:51.07787858 +0000 UTC m=+690.499755056" watchObservedRunningTime="2026-07-10 16:47:51.078399482 +0000 UTC m=+690.500275947" Jul 10 16:47:51.080437 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:51.080411 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wrdwj\" is forbidden: User \"system:node:ip-10-0-134-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-41.ec2.internal' and this object" podUID="fb4daf3d-5927-44b3-9452-85e4c6f6c94b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" Jul 10 16:47:51.324597 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:51.324507 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2ftlq\" is forbidden: User \"system:node:ip-10-0-134-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-41.ec2.internal' and this object" podUID="3f07ee45-4540-4698-821a-c56c8a3382b8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ftlq" Jul 10 16:47:51.325452 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.325432 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f07ee45-4540-4698-821a-c56c8a3382b8" path="/var/lib/kubelet/pods/3f07ee45-4540-4698-821a-c56c8a3382b8/volumes" Jul 10 16:47:51.325783 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:51.325769 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb4daf3d-5927-44b3-9452-85e4c6f6c94b" path="/var/lib/kubelet/pods/fb4daf3d-5927-44b3-9452-85e4c6f6c94b/volumes" Jul 10 16:47:51.326566 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:51.326545 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wrdwj\" is forbidden: User \"system:node:ip-10-0-134-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-41.ec2.internal' and this object" podUID="fb4daf3d-5927-44b3-9452-85e4c6f6c94b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wrdwj" Jul 10 16:47:53.631573 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:53.631542 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2"] Jul 10 16:47:53.632037 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:53.631709 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2" podUID="ddc62214-6483-4a44-ae77-871cc8cb7d42" containerName="manager" containerID="cri-o://f42405332bf43b3b8bd819d10103cded21d79dc32853f9c60fadfbc9a25418da" gracePeriod=10 Jul 10 16:47:53.884911 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:53.884856 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2" Jul 10 16:47:53.886567 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:53.886549 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxw8h\" (UniqueName: \"kubernetes.io/projected/ddc62214-6483-4a44-ae77-871cc8cb7d42-kube-api-access-mxw8h\") pod \"ddc62214-6483-4a44-ae77-871cc8cb7d42\" (UID: \"ddc62214-6483-4a44-ae77-871cc8cb7d42\") " Jul 10 16:47:53.886642 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:53.886631 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddc62214-6483-4a44-ae77-871cc8cb7d42-extensions-socket-volume\") pod \"ddc62214-6483-4a44-ae77-871cc8cb7d42\" (UID: \"ddc62214-6483-4a44-ae77-871cc8cb7d42\") " Jul 10 16:47:53.887037 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:53.887016 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc62214-6483-4a44-ae77-871cc8cb7d42-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ddc62214-6483-4a44-ae77-871cc8cb7d42" (UID: "ddc62214-6483-4a44-ae77-871cc8cb7d42"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:47:53.888527 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:53.888498 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc62214-6483-4a44-ae77-871cc8cb7d42-kube-api-access-mxw8h" (OuterVolumeSpecName: "kube-api-access-mxw8h") pod "ddc62214-6483-4a44-ae77-871cc8cb7d42" (UID: "ddc62214-6483-4a44-ae77-871cc8cb7d42"). InnerVolumeSpecName "kube-api-access-mxw8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:47:53.987214 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:53.987178 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddc62214-6483-4a44-ae77-871cc8cb7d42-extensions-socket-volume\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:47:53.987214 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:53.987206 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxw8h\" (UniqueName: \"kubernetes.io/projected/ddc62214-6483-4a44-ae77-871cc8cb7d42-kube-api-access-mxw8h\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:47:54.040369 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:54.040336 2585 generic.go:356] "Generic (PLEG): container finished" podID="ddc62214-6483-4a44-ae77-871cc8cb7d42" containerID="f42405332bf43b3b8bd819d10103cded21d79dc32853f9c60fadfbc9a25418da" exitCode=0 Jul 10 16:47:54.040542 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:54.040380 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2" Jul 10 16:47:54.040542 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:54.040397 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2" event={"ID":"ddc62214-6483-4a44-ae77-871cc8cb7d42","Type":"ContainerDied","Data":"f42405332bf43b3b8bd819d10103cded21d79dc32853f9c60fadfbc9a25418da"} Jul 10 16:47:54.040542 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:54.040445 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2" event={"ID":"ddc62214-6483-4a44-ae77-871cc8cb7d42","Type":"ContainerDied","Data":"ae8557377bc84891ba93643a570d8cb8f9c26a7a221ff54cbce68f7a91a23e25"} Jul 10 16:47:54.040542 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:54.040466 2585 scope.go:117] "RemoveContainer" containerID="f42405332bf43b3b8bd819d10103cded21d79dc32853f9c60fadfbc9a25418da" Jul 10 16:47:54.049780 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:54.049764 2585 scope.go:117] "RemoveContainer" containerID="f42405332bf43b3b8bd819d10103cded21d79dc32853f9c60fadfbc9a25418da" Jul 10 16:47:54.050038 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:47:54.050021 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f42405332bf43b3b8bd819d10103cded21d79dc32853f9c60fadfbc9a25418da\": container with ID starting with f42405332bf43b3b8bd819d10103cded21d79dc32853f9c60fadfbc9a25418da not found: ID does not exist" containerID="f42405332bf43b3b8bd819d10103cded21d79dc32853f9c60fadfbc9a25418da" Jul 10 16:47:54.050085 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:54.050046 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f42405332bf43b3b8bd819d10103cded21d79dc32853f9c60fadfbc9a25418da"} err="failed to get container status \"f42405332bf43b3b8bd819d10103cded21d79dc32853f9c60fadfbc9a25418da\": rpc error: code = NotFound desc = could not find container \"f42405332bf43b3b8bd819d10103cded21d79dc32853f9c60fadfbc9a25418da\": container with ID starting with f42405332bf43b3b8bd819d10103cded21d79dc32853f9c60fadfbc9a25418da not found: ID does not exist" Jul 10 16:47:54.069657 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:54.069628 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2"] Jul 10 16:47:54.076356 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:54.076331 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bhwn2"] Jul 10 16:47:55.326333 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:47:55.326300 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddc62214-6483-4a44-ae77-871cc8cb7d42" path="/var/lib/kubelet/pods/ddc62214-6483-4a44-ae77-871cc8cb7d42/volumes" Jul 10 16:48:02.031990 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:02.031959 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4fkhd" Jul 10 16:48:09.949845 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:09.949805 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv"] Jul 10 16:48:09.950273 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:09.950200 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddc62214-6483-4a44-ae77-871cc8cb7d42" containerName="manager" Jul 10 16:48:09.950273 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:09.950215 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddc62214-6483-4a44-ae77-871cc8cb7d42" containerName="manager" Jul 10 16:48:09.950344 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:09.950297 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddc62214-6483-4a44-ae77-871cc8cb7d42" containerName="manager" Jul 10 16:48:10.132425 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.132392 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv"] Jul 10 16:48:10.132595 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.132503 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.135512 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.135487 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wdm5t\"" Jul 10 16:48:10.227089 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.226995 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00adb9cb-ed88-4eb5-bed7-63f59639ae86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.227089 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.227037 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00adb9cb-ed88-4eb5-bed7-63f59639ae86-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.227321 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.227101 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00adb9cb-ed88-4eb5-bed7-63f59639ae86-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.227321 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.227140 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00adb9cb-ed88-4eb5-bed7-63f59639ae86-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.227321 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.227167 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00adb9cb-ed88-4eb5-bed7-63f59639ae86-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.227321 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.227207 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00adb9cb-ed88-4eb5-bed7-63f59639ae86-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.227321 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.227237 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00adb9cb-ed88-4eb5-bed7-63f59639ae86-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.227321 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.227272 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00adb9cb-ed88-4eb5-bed7-63f59639ae86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.227321 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.227303 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnsr5\" (UniqueName: \"kubernetes.io/projected/00adb9cb-ed88-4eb5-bed7-63f59639ae86-kube-api-access-tnsr5\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.328793 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.328728 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00adb9cb-ed88-4eb5-bed7-63f59639ae86-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.328994 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.328805 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00adb9cb-ed88-4eb5-bed7-63f59639ae86-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.328994 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.328830 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00adb9cb-ed88-4eb5-bed7-63f59639ae86-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.328994 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.328862 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00adb9cb-ed88-4eb5-bed7-63f59639ae86-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.328994 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.328881 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00adb9cb-ed88-4eb5-bed7-63f59639ae86-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.328994 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.328917 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00adb9cb-ed88-4eb5-bed7-63f59639ae86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.328994 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.328951 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnsr5\" (UniqueName: \"kubernetes.io/projected/00adb9cb-ed88-4eb5-bed7-63f59639ae86-kube-api-access-tnsr5\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.329314 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.329022 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00adb9cb-ed88-4eb5-bed7-63f59639ae86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.329314 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.329052 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00adb9cb-ed88-4eb5-bed7-63f59639ae86-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.329314 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.329206 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00adb9cb-ed88-4eb5-bed7-63f59639ae86-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.329422 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.329334 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00adb9cb-ed88-4eb5-bed7-63f59639ae86-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.329422 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.329384 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00adb9cb-ed88-4eb5-bed7-63f59639ae86-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.329496 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.329461 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00adb9cb-ed88-4eb5-bed7-63f59639ae86-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.329806 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.329788 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00adb9cb-ed88-4eb5-bed7-63f59639ae86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.331213 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.331186 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00adb9cb-ed88-4eb5-bed7-63f59639ae86-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.331312 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.331270 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00adb9cb-ed88-4eb5-bed7-63f59639ae86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.338022 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.337998 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00adb9cb-ed88-4eb5-bed7-63f59639ae86-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.338186 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.338170 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnsr5\" (UniqueName: \"kubernetes.io/projected/00adb9cb-ed88-4eb5-bed7-63f59639ae86-kube-api-access-tnsr5\") pod \"maas-default-gateway-openshift-default-56654dcf7-cvchv\" (UID: \"00adb9cb-ed88-4eb5-bed7-63f59639ae86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.443690 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.443657 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:10.611068 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.611037 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv"] Jul 10 16:48:10.612354 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:48:10.612322 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00adb9cb_ed88_4eb5_bed7_63f59639ae86.slice/crio-ba84fb80b61f875ccc22d0ee38075c06cd69c306a8f864b025fc05216b1f8378 WatchSource:0}: Error finding container ba84fb80b61f875ccc22d0ee38075c06cd69c306a8f864b025fc05216b1f8378: Status 404 returned error can't find the container with id ba84fb80b61f875ccc22d0ee38075c06cd69c306a8f864b025fc05216b1f8378 Jul 10 16:48:10.614707 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.614674 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 16:48:10.614865 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.614761 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 16:48:10.614865 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:10.614805 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 16:48:11.137123 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:11.137084 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" event={"ID":"00adb9cb-ed88-4eb5-bed7-63f59639ae86","Type":"ContainerStarted","Data":"481b7e9c047b4d146a7162cb088f78aa85f7a53f515f00af150f75ca0d21638f"} Jul 10 16:48:11.137525 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:11.137147 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" event={"ID":"00adb9cb-ed88-4eb5-bed7-63f59639ae86","Type":"ContainerStarted","Data":"ba84fb80b61f875ccc22d0ee38075c06cd69c306a8f864b025fc05216b1f8378"} Jul 10 16:48:11.162070 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:11.162019 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" podStartSLOduration=2.16200398 podStartE2EDuration="2.16200398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:48:11.160004881 +0000 UTC m=+710.581881369" watchObservedRunningTime="2026-07-10 16:48:11.16200398 +0000 UTC m=+710.583880445" Jul 10 16:48:11.444373 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:11.444278 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:11.449038 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:11.449015 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:12.142306 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:12.142271 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:12.143355 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:12.143335 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:48:32.645771 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:32.645674 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:48:32.698189 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:32.698157 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:48:32.698189 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:32.698195 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:48:32.698365 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:32.698265 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-6nmhd" Jul 10 16:48:32.700980 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:32.700963 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 16:48:32.825286 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:32.825259 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4b26840e-4e20-4666-bacc-6deb8f4e0f37-config-file\") pod \"limitador-limitador-54d6d988d5-6nmhd\" (UID: \"4b26840e-4e20-4666-bacc-6deb8f4e0f37\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6nmhd" Jul 10 16:48:32.825440 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:32.825347 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49f4p\" (UniqueName: \"kubernetes.io/projected/4b26840e-4e20-4666-bacc-6deb8f4e0f37-kube-api-access-49f4p\") pod \"limitador-limitador-54d6d988d5-6nmhd\" (UID: \"4b26840e-4e20-4666-bacc-6deb8f4e0f37\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6nmhd" Jul 10 16:48:32.925828 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:32.925764 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49f4p\" (UniqueName: \"kubernetes.io/projected/4b26840e-4e20-4666-bacc-6deb8f4e0f37-kube-api-access-49f4p\") pod \"limitador-limitador-54d6d988d5-6nmhd\" (UID: \"4b26840e-4e20-4666-bacc-6deb8f4e0f37\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6nmhd" Jul 10 16:48:32.925934 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:32.925823 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4b26840e-4e20-4666-bacc-6deb8f4e0f37-config-file\") pod \"limitador-limitador-54d6d988d5-6nmhd\" (UID: \"4b26840e-4e20-4666-bacc-6deb8f4e0f37\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6nmhd" Jul 10 16:48:32.926405 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:32.926387 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4b26840e-4e20-4666-bacc-6deb8f4e0f37-config-file\") pod \"limitador-limitador-54d6d988d5-6nmhd\" (UID: \"4b26840e-4e20-4666-bacc-6deb8f4e0f37\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6nmhd" Jul 10 16:48:32.934061 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:32.934044 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49f4p\" (UniqueName: \"kubernetes.io/projected/4b26840e-4e20-4666-bacc-6deb8f4e0f37-kube-api-access-49f4p\") pod \"limitador-limitador-54d6d988d5-6nmhd\" (UID: \"4b26840e-4e20-4666-bacc-6deb8f4e0f37\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6nmhd" Jul 10 16:48:33.008542 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:33.008515 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-6nmhd" Jul 10 16:48:33.140814 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:33.140788 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:48:33.145031 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:48:33.145003 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b26840e_4e20_4666_bacc_6deb8f4e0f37.slice/crio-9cdc1c4db34e1530df74575bcf316c2ab7f19fceaaef07fc415005a932ea4b3c WatchSource:0}: Error finding container 9cdc1c4db34e1530df74575bcf316c2ab7f19fceaaef07fc415005a932ea4b3c: Status 404 returned error can't find the container with id 9cdc1c4db34e1530df74575bcf316c2ab7f19fceaaef07fc415005a932ea4b3c Jul 10 16:48:33.147349 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:33.147333 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 16:48:33.181808 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:33.181718 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-vvjw4"] Jul 10 16:48:33.186509 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:33.186489 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vvjw4" Jul 10 16:48:33.189526 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:33.189505 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xzc5g\"" Jul 10 16:48:33.189867 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:33.189849 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vvjw4"] Jul 10 16:48:33.251956 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:33.251916 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-6nmhd" event={"ID":"4b26840e-4e20-4666-bacc-6deb8f4e0f37","Type":"ContainerStarted","Data":"9cdc1c4db34e1530df74575bcf316c2ab7f19fceaaef07fc415005a932ea4b3c"} Jul 10 16:48:33.330371 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:33.330341 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvjxh\" (UniqueName: \"kubernetes.io/projected/0ac35e7c-8ccf-4197-a6e1-10cf269f72db-kube-api-access-kvjxh\") pod \"authorino-75d556bd77-vvjw4\" (UID: \"0ac35e7c-8ccf-4197-a6e1-10cf269f72db\") " pod="kuadrant-system/authorino-75d556bd77-vvjw4" Jul 10 16:48:33.431139 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:33.431101 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvjxh\" (UniqueName: \"kubernetes.io/projected/0ac35e7c-8ccf-4197-a6e1-10cf269f72db-kube-api-access-kvjxh\") pod \"authorino-75d556bd77-vvjw4\" (UID: \"0ac35e7c-8ccf-4197-a6e1-10cf269f72db\") " pod="kuadrant-system/authorino-75d556bd77-vvjw4" Jul 10 16:48:33.439018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:33.438956 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvjxh\" (UniqueName: \"kubernetes.io/projected/0ac35e7c-8ccf-4197-a6e1-10cf269f72db-kube-api-access-kvjxh\") pod \"authorino-75d556bd77-vvjw4\" (UID: \"0ac35e7c-8ccf-4197-a6e1-10cf269f72db\") " pod="kuadrant-system/authorino-75d556bd77-vvjw4" Jul 10 16:48:33.496383 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:33.496348 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vvjw4" Jul 10 16:48:33.625454 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:33.625425 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vvjw4"] Jul 10 16:48:33.626647 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:48:33.626618 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ac35e7c_8ccf_4197_a6e1_10cf269f72db.slice/crio-9f48fe52b676284ef4ad576f67f93f2078bd7e0ffa1da2fa9afc4d0d9cd2c720 WatchSource:0}: Error finding container 9f48fe52b676284ef4ad576f67f93f2078bd7e0ffa1da2fa9afc4d0d9cd2c720: Status 404 returned error can't find the container with id 9f48fe52b676284ef4ad576f67f93f2078bd7e0ffa1da2fa9afc4d0d9cd2c720 Jul 10 16:48:34.263478 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:34.263410 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vvjw4" event={"ID":"0ac35e7c-8ccf-4197-a6e1-10cf269f72db","Type":"ContainerStarted","Data":"9f48fe52b676284ef4ad576f67f93f2078bd7e0ffa1da2fa9afc4d0d9cd2c720"} Jul 10 16:48:38.293193 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:38.293085 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-6nmhd" event={"ID":"4b26840e-4e20-4666-bacc-6deb8f4e0f37","Type":"ContainerStarted","Data":"3eb5f59aaee1c6fdc29642e32408235f78e046f1ba21f339adbf0d4663cabcdc"} Jul 10 16:48:38.293193 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:38.293169 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-6nmhd" Jul 10 16:48:38.294549 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:38.294526 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vvjw4" event={"ID":"0ac35e7c-8ccf-4197-a6e1-10cf269f72db","Type":"ContainerStarted","Data":"8fbd3d13951fde72c4d7ebd72ce87e58935d4a5dc09e267ffecf168ada9bb343"} Jul 10 16:48:38.309863 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:38.309819 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-6nmhd" podStartSLOduration=1.597552033 podStartE2EDuration="6.309804484s" totalImagesPullingTime="4.712252451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:48:38.308338991 +0000 UTC m=+737.730215474" watchObservedRunningTime="2026-07-10 16:48:38.309804484 +0000 UTC m=+737.731680950" Jul 10 16:48:38.324510 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:38.324458 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-vvjw4" podStartSLOduration=1.092694665 podStartE2EDuration="5.324443374s" totalImagesPullingTime="4.231748709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:48:38.322730368 +0000 UTC m=+737.744606835" watchObservedRunningTime="2026-07-10 16:48:38.324443374 +0000 UTC m=+737.746319840" Jul 10 16:48:49.301602 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:48:49.301569 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-6nmhd" Jul 10 16:49:07.944028 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:07.943997 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-845f7d5bd-4jch6"] Jul 10 16:49:07.949301 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:07.949277 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-845f7d5bd-4jch6" Jul 10 16:49:07.952419 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:07.952389 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 10 16:49:07.955027 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:07.955002 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-845f7d5bd-4jch6"] Jul 10 16:49:08.046546 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:08.046510 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/467b0c34-6f79-4ea9-8bce-86b53578b73f-tls-cert\") pod \"authorino-845f7d5bd-4jch6\" (UID: \"467b0c34-6f79-4ea9-8bce-86b53578b73f\") " pod="kuadrant-system/authorino-845f7d5bd-4jch6" Jul 10 16:49:08.046728 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:08.046557 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phwd2\" (UniqueName: \"kubernetes.io/projected/467b0c34-6f79-4ea9-8bce-86b53578b73f-kube-api-access-phwd2\") pod \"authorino-845f7d5bd-4jch6\" (UID: \"467b0c34-6f79-4ea9-8bce-86b53578b73f\") " pod="kuadrant-system/authorino-845f7d5bd-4jch6" Jul 10 16:49:08.147887 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:08.147849 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/467b0c34-6f79-4ea9-8bce-86b53578b73f-tls-cert\") pod \"authorino-845f7d5bd-4jch6\" (UID: \"467b0c34-6f79-4ea9-8bce-86b53578b73f\") " pod="kuadrant-system/authorino-845f7d5bd-4jch6" Jul 10 16:49:08.148030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:08.147899 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phwd2\" (UniqueName: \"kubernetes.io/projected/467b0c34-6f79-4ea9-8bce-86b53578b73f-kube-api-access-phwd2\") pod \"authorino-845f7d5bd-4jch6\" (UID: \"467b0c34-6f79-4ea9-8bce-86b53578b73f\") " pod="kuadrant-system/authorino-845f7d5bd-4jch6" Jul 10 16:49:08.150243 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:08.150220 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/467b0c34-6f79-4ea9-8bce-86b53578b73f-tls-cert\") pod \"authorino-845f7d5bd-4jch6\" (UID: \"467b0c34-6f79-4ea9-8bce-86b53578b73f\") " pod="kuadrant-system/authorino-845f7d5bd-4jch6" Jul 10 16:49:08.156440 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:08.156414 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phwd2\" (UniqueName: \"kubernetes.io/projected/467b0c34-6f79-4ea9-8bce-86b53578b73f-kube-api-access-phwd2\") pod \"authorino-845f7d5bd-4jch6\" (UID: \"467b0c34-6f79-4ea9-8bce-86b53578b73f\") " pod="kuadrant-system/authorino-845f7d5bd-4jch6" Jul 10 16:49:08.259666 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:08.259581 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-845f7d5bd-4jch6" Jul 10 16:49:08.404318 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:08.404292 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-845f7d5bd-4jch6"] Jul 10 16:49:08.406382 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:49:08.406353 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod467b0c34_6f79_4ea9_8bce_86b53578b73f.slice/crio-9b28fa3d50d2d3c62467fa03a265e4bdb6397f050fb1b4f7234f64c3a602d0eb WatchSource:0}: Error finding container 9b28fa3d50d2d3c62467fa03a265e4bdb6397f050fb1b4f7234f64c3a602d0eb: Status 404 returned error can't find the container with id 9b28fa3d50d2d3c62467fa03a265e4bdb6397f050fb1b4f7234f64c3a602d0eb Jul 10 16:49:08.452545 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:08.452517 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-845f7d5bd-4jch6" event={"ID":"467b0c34-6f79-4ea9-8bce-86b53578b73f","Type":"ContainerStarted","Data":"9b28fa3d50d2d3c62467fa03a265e4bdb6397f050fb1b4f7234f64c3a602d0eb"} Jul 10 16:49:09.459216 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:09.459174 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-845f7d5bd-4jch6" event={"ID":"467b0c34-6f79-4ea9-8bce-86b53578b73f","Type":"ContainerStarted","Data":"02221950de729602e0945bef7b35ba6610f3f0b04496ff36296580e6dbc2eaf9"} Jul 10 16:49:09.476425 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:09.476355 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-845f7d5bd-4jch6" podStartSLOduration=2.154895024 podStartE2EDuration="2.47633343s" totalImagesPullingTime="321.438406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:49:09.473405713 +0000 UTC m=+768.895282180" watchObservedRunningTime="2026-07-10 16:49:09.47633343 +0000 UTC m=+768.898209902" Jul 10 16:49:09.499549 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:09.499519 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vvjw4"] Jul 10 16:49:09.499720 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:09.499697 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-vvjw4" podUID="0ac35e7c-8ccf-4197-a6e1-10cf269f72db" containerName="authorino" containerID="cri-o://8fbd3d13951fde72c4d7ebd72ce87e58935d4a5dc09e267ffecf168ada9bb343" gracePeriod=30 Jul 10 16:49:09.776533 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:09.776504 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vvjw4" Jul 10 16:49:09.863975 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:09.863945 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvjxh\" (UniqueName: \"kubernetes.io/projected/0ac35e7c-8ccf-4197-a6e1-10cf269f72db-kube-api-access-kvjxh\") pod \"0ac35e7c-8ccf-4197-a6e1-10cf269f72db\" (UID: \"0ac35e7c-8ccf-4197-a6e1-10cf269f72db\") " Jul 10 16:49:09.865973 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:09.865946 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac35e7c-8ccf-4197-a6e1-10cf269f72db-kube-api-access-kvjxh" (OuterVolumeSpecName: "kube-api-access-kvjxh") pod "0ac35e7c-8ccf-4197-a6e1-10cf269f72db" (UID: "0ac35e7c-8ccf-4197-a6e1-10cf269f72db"). InnerVolumeSpecName "kube-api-access-kvjxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:49:09.965564 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:09.965532 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvjxh\" (UniqueName: \"kubernetes.io/projected/0ac35e7c-8ccf-4197-a6e1-10cf269f72db-kube-api-access-kvjxh\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:49:10.465584 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:10.465543 2585 generic.go:356] "Generic (PLEG): container finished" podID="0ac35e7c-8ccf-4197-a6e1-10cf269f72db" containerID="8fbd3d13951fde72c4d7ebd72ce87e58935d4a5dc09e267ffecf168ada9bb343" exitCode=0 Jul 10 16:49:10.465584 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:10.465588 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vvjw4" Jul 10 16:49:10.466128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:10.465623 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vvjw4" event={"ID":"0ac35e7c-8ccf-4197-a6e1-10cf269f72db","Type":"ContainerDied","Data":"8fbd3d13951fde72c4d7ebd72ce87e58935d4a5dc09e267ffecf168ada9bb343"} Jul 10 16:49:10.466128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:10.465679 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vvjw4" event={"ID":"0ac35e7c-8ccf-4197-a6e1-10cf269f72db","Type":"ContainerDied","Data":"9f48fe52b676284ef4ad576f67f93f2078bd7e0ffa1da2fa9afc4d0d9cd2c720"} Jul 10 16:49:10.466128 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:10.465704 2585 scope.go:117] "RemoveContainer" containerID="8fbd3d13951fde72c4d7ebd72ce87e58935d4a5dc09e267ffecf168ada9bb343" Jul 10 16:49:10.475129 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:10.475110 2585 scope.go:117] "RemoveContainer" containerID="8fbd3d13951fde72c4d7ebd72ce87e58935d4a5dc09e267ffecf168ada9bb343" Jul 10 16:49:10.475388 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:49:10.475371 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fbd3d13951fde72c4d7ebd72ce87e58935d4a5dc09e267ffecf168ada9bb343\": container with ID starting with 8fbd3d13951fde72c4d7ebd72ce87e58935d4a5dc09e267ffecf168ada9bb343 not found: ID does not exist" containerID="8fbd3d13951fde72c4d7ebd72ce87e58935d4a5dc09e267ffecf168ada9bb343" Jul 10 16:49:10.475460 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:10.475395 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fbd3d13951fde72c4d7ebd72ce87e58935d4a5dc09e267ffecf168ada9bb343"} err="failed to get container status \"8fbd3d13951fde72c4d7ebd72ce87e58935d4a5dc09e267ffecf168ada9bb343\": rpc error: code = NotFound desc = could not find container \"8fbd3d13951fde72c4d7ebd72ce87e58935d4a5dc09e267ffecf168ada9bb343\": container with ID starting with 8fbd3d13951fde72c4d7ebd72ce87e58935d4a5dc09e267ffecf168ada9bb343 not found: ID does not exist" Jul 10 16:49:10.487306 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:10.487282 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vvjw4"] Jul 10 16:49:10.492610 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:10.492591 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vvjw4"] Jul 10 16:49:11.325420 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:11.325390 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ac35e7c-8ccf-4197-a6e1-10cf269f72db" path="/var/lib/kubelet/pods/0ac35e7c-8ccf-4197-a6e1-10cf269f72db/volumes" Jul 10 16:49:50.486824 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:50.486727 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:49:57.965136 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:57.965098 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cvchv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": read tcp 10.133.0.2:36790->10.133.0.42:15021: read: connection reset by peer" start-of-body= Jul 10 16:49:57.965557 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:57.965196 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" podUID="00adb9cb-ed88-4eb5-bed7-63f59639ae86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": read tcp 10.133.0.2:36790->10.133.0.42:15021: read: connection reset by peer" Jul 10 16:49:58.731755 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:58.731719 2585 generic.go:356] "Generic (PLEG): container finished" podID="00adb9cb-ed88-4eb5-bed7-63f59639ae86" containerID="481b7e9c047b4d146a7162cb088f78aa85f7a53f515f00af150f75ca0d21638f" exitCode=0 Jul 10 16:49:58.731936 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:58.731805 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" event={"ID":"00adb9cb-ed88-4eb5-bed7-63f59639ae86","Type":"ContainerDied","Data":"481b7e9c047b4d146a7162cb088f78aa85f7a53f515f00af150f75ca0d21638f"} Jul 10 16:49:58.732226 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:58.732211 2585 scope.go:117] "RemoveContainer" containerID="481b7e9c047b4d146a7162cb088f78aa85f7a53f515f00af150f75ca0d21638f" Jul 10 16:49:58.733189 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:58.733158 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 16:49:58.733292 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:58.733246 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 16:49:58.733354 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:58.733299 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 16:49:59.742799 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:49:59.742730 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" event={"ID":"00adb9cb-ed88-4eb5-bed7-63f59639ae86","Type":"ContainerStarted","Data":"7a0880451f029d1472695808575b429c1e07561bef73bb0cd0e4b6674f5d4f4a"} Jul 10 16:50:00.444789 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:00.444734 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:50:00.445018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:00.444817 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cvchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 10 16:50:00.445018 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:00.444899 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" podUID="00adb9cb-ed88-4eb5-bed7-63f59639ae86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 10 16:50:01.444523 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:01.444475 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cvchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 10 16:50:01.445050 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:01.444545 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" podUID="00adb9cb-ed88-4eb5-bed7-63f59639ae86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 10 16:50:02.444642 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:02.444598 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cvchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 10 16:50:02.445695 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:02.445257 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" podUID="00adb9cb-ed88-4eb5-bed7-63f59639ae86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 10 16:50:03.444483 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:03.444438 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cvchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 10 16:50:03.444701 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:03.444514 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" podUID="00adb9cb-ed88-4eb5-bed7-63f59639ae86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 10 16:50:04.447706 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:04.447671 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:50:04.448172 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:04.447936 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:50:04.448563 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:04.448546 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cvchv" Jul 10 16:50:07.083565 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:07.083532 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:50:13.199654 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:13.199616 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:50:16.995863 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:16.995831 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:50:33.093633 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:33.093595 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:50:41.693362 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:41.693304 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:50:47.107030 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.106985 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29"] Jul 10 16:50:47.107838 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.107813 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ac35e7c-8ccf-4197-a6e1-10cf269f72db" containerName="authorino" Jul 10 16:50:47.107838 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.107839 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ac35e7c-8ccf-4197-a6e1-10cf269f72db" containerName="authorino" Jul 10 16:50:47.108052 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.108039 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ac35e7c-8ccf-4197-a6e1-10cf269f72db" containerName="authorino" Jul 10 16:50:47.111669 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.111644 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.115895 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.115871 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kjfph\"" Jul 10 16:50:47.116054 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.115872 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 16:50:47.116432 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.116415 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 16:50:47.117646 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.117621 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 10 16:50:47.121263 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.121238 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29"] Jul 10 16:50:47.254564 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.254531 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/356f50aa-3017-49a8-8733-c4dfa7ec8ee7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29\" (UID: \"356f50aa-3017-49a8-8733-c4dfa7ec8ee7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.254564 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.254569 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd55f\" (UniqueName: \"kubernetes.io/projected/356f50aa-3017-49a8-8733-c4dfa7ec8ee7-kube-api-access-cd55f\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29\" (UID: \"356f50aa-3017-49a8-8733-c4dfa7ec8ee7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.254857 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.254609 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/356f50aa-3017-49a8-8733-c4dfa7ec8ee7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29\" (UID: \"356f50aa-3017-49a8-8733-c4dfa7ec8ee7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.254857 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.254738 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/356f50aa-3017-49a8-8733-c4dfa7ec8ee7-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29\" (UID: \"356f50aa-3017-49a8-8733-c4dfa7ec8ee7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.254857 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.254831 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/356f50aa-3017-49a8-8733-c4dfa7ec8ee7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29\" (UID: \"356f50aa-3017-49a8-8733-c4dfa7ec8ee7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.254962 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.254869 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/356f50aa-3017-49a8-8733-c4dfa7ec8ee7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29\" (UID: \"356f50aa-3017-49a8-8733-c4dfa7ec8ee7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.355831 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.355781 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/356f50aa-3017-49a8-8733-c4dfa7ec8ee7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29\" (UID: \"356f50aa-3017-49a8-8733-c4dfa7ec8ee7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.356011 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.355838 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd55f\" (UniqueName: \"kubernetes.io/projected/356f50aa-3017-49a8-8733-c4dfa7ec8ee7-kube-api-access-cd55f\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29\" (UID: \"356f50aa-3017-49a8-8733-c4dfa7ec8ee7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.356011 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.355893 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/356f50aa-3017-49a8-8733-c4dfa7ec8ee7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29\" (UID: \"356f50aa-3017-49a8-8733-c4dfa7ec8ee7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.356011 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.355938 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/356f50aa-3017-49a8-8733-c4dfa7ec8ee7-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29\" (UID: \"356f50aa-3017-49a8-8733-c4dfa7ec8ee7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.356011 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.355996 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/356f50aa-3017-49a8-8733-c4dfa7ec8ee7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29\" (UID: \"356f50aa-3017-49a8-8733-c4dfa7ec8ee7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.356256 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.356035 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/356f50aa-3017-49a8-8733-c4dfa7ec8ee7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29\" (UID: \"356f50aa-3017-49a8-8733-c4dfa7ec8ee7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.356370 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.356348 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/356f50aa-3017-49a8-8733-c4dfa7ec8ee7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29\" (UID: \"356f50aa-3017-49a8-8733-c4dfa7ec8ee7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.356436 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.356356 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/356f50aa-3017-49a8-8733-c4dfa7ec8ee7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29\" (UID: \"356f50aa-3017-49a8-8733-c4dfa7ec8ee7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.356436 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.356402 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/356f50aa-3017-49a8-8733-c4dfa7ec8ee7-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29\" (UID: \"356f50aa-3017-49a8-8733-c4dfa7ec8ee7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.358248 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.358187 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/356f50aa-3017-49a8-8733-c4dfa7ec8ee7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29\" (UID: \"356f50aa-3017-49a8-8733-c4dfa7ec8ee7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.358458 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.358440 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/356f50aa-3017-49a8-8733-c4dfa7ec8ee7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29\" (UID: \"356f50aa-3017-49a8-8733-c4dfa7ec8ee7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.365684 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.365657 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd55f\" (UniqueName: \"kubernetes.io/projected/356f50aa-3017-49a8-8733-c4dfa7ec8ee7-kube-api-access-cd55f\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29\" (UID: \"356f50aa-3017-49a8-8733-c4dfa7ec8ee7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.422639 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.422598 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:47.569467 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:47.569440 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29"] Jul 10 16:50:47.570834 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:50:47.570798 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod356f50aa_3017_49a8_8733_c4dfa7ec8ee7.slice/crio-936adb5d01d855294e8248e1e5397e2e907dfdb97456d155f7eea11cae85674f WatchSource:0}: Error finding container 936adb5d01d855294e8248e1e5397e2e907dfdb97456d155f7eea11cae85674f: Status 404 returned error can't find the container with id 936adb5d01d855294e8248e1e5397e2e907dfdb97456d155f7eea11cae85674f Jul 10 16:50:48.013501 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:48.013429 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" event={"ID":"356f50aa-3017-49a8-8733-c4dfa7ec8ee7","Type":"ContainerStarted","Data":"936adb5d01d855294e8248e1e5397e2e907dfdb97456d155f7eea11cae85674f"} Jul 10 16:50:49.020837 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:49.020797 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" event={"ID":"356f50aa-3017-49a8-8733-c4dfa7ec8ee7","Type":"ContainerStarted","Data":"20d6b18be4e145c324022a7bb96135c2d383ae680aa05e61f0841a904e7f04ae"} Jul 10 16:50:49.045153 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:49.045097 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" podStartSLOduration=1.024220515 podStartE2EDuration="2.04507854s" totalImagesPullingTime="1.020858025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:50:49.042768221 +0000 UTC m=+868.464644680" watchObservedRunningTime="2026-07-10 16:50:49.04507854 +0000 UTC m=+868.466955006" Jul 10 16:50:51.218112 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:51.218066 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:50:57.422736 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:57.422702 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:57.422736 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:57.422772 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:57.435572 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:57.435546 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:50:58.081475 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:50:58.081438 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29" Jul 10 16:51:21.031945 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:21.031866 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 16:51:21.033696 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:21.033665 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 16:51:46.739787 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:46.739723 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-96d6c78b5-2tdp8"] Jul 10 16:51:46.743397 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:46.743377 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-96d6c78b5-2tdp8" Jul 10 16:51:46.753957 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:46.753928 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-96d6c78b5-2tdp8"] Jul 10 16:51:46.913612 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:46.913553 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssq29\" (UniqueName: \"kubernetes.io/projected/c64833da-5ba0-4a94-a050-88c6907421fa-kube-api-access-ssq29\") pod \"authorino-96d6c78b5-2tdp8\" (UID: \"c64833da-5ba0-4a94-a050-88c6907421fa\") " pod="kuadrant-system/authorino-96d6c78b5-2tdp8" Jul 10 16:51:46.913845 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:46.913718 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c64833da-5ba0-4a94-a050-88c6907421fa-tls-cert\") pod \"authorino-96d6c78b5-2tdp8\" (UID: \"c64833da-5ba0-4a94-a050-88c6907421fa\") " pod="kuadrant-system/authorino-96d6c78b5-2tdp8" Jul 10 16:51:47.014629 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:47.014535 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c64833da-5ba0-4a94-a050-88c6907421fa-tls-cert\") pod \"authorino-96d6c78b5-2tdp8\" (UID: \"c64833da-5ba0-4a94-a050-88c6907421fa\") " pod="kuadrant-system/authorino-96d6c78b5-2tdp8" Jul 10 16:51:47.014629 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:47.014623 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssq29\" (UniqueName: \"kubernetes.io/projected/c64833da-5ba0-4a94-a050-88c6907421fa-kube-api-access-ssq29\") pod \"authorino-96d6c78b5-2tdp8\" (UID: \"c64833da-5ba0-4a94-a050-88c6907421fa\") " pod="kuadrant-system/authorino-96d6c78b5-2tdp8" Jul 10 16:51:47.017177 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:47.017150 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c64833da-5ba0-4a94-a050-88c6907421fa-tls-cert\") pod \"authorino-96d6c78b5-2tdp8\" (UID: \"c64833da-5ba0-4a94-a050-88c6907421fa\") " pod="kuadrant-system/authorino-96d6c78b5-2tdp8" Jul 10 16:51:47.025590 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:47.025564 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssq29\" (UniqueName: \"kubernetes.io/projected/c64833da-5ba0-4a94-a050-88c6907421fa-kube-api-access-ssq29\") pod \"authorino-96d6c78b5-2tdp8\" (UID: \"c64833da-5ba0-4a94-a050-88c6907421fa\") " pod="kuadrant-system/authorino-96d6c78b5-2tdp8" Jul 10 16:51:47.054740 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:47.054710 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-96d6c78b5-2tdp8" Jul 10 16:51:47.225978 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:47.225955 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-96d6c78b5-2tdp8"] Jul 10 16:51:47.228946 ip-10-0-134-41 kubenswrapper[2585]: W0710 16:51:47.228904 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc64833da_5ba0_4a94_a050_88c6907421fa.slice/crio-4f3381688e59f3ae2f341f269245572cf3dae639f0636d6952f921012a7d0393 WatchSource:0}: Error finding container 4f3381688e59f3ae2f341f269245572cf3dae639f0636d6952f921012a7d0393: Status 404 returned error can't find the container with id 4f3381688e59f3ae2f341f269245572cf3dae639f0636d6952f921012a7d0393 Jul 10 16:51:47.333461 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:47.333374 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-96d6c78b5-2tdp8" event={"ID":"c64833da-5ba0-4a94-a050-88c6907421fa","Type":"ContainerStarted","Data":"4f3381688e59f3ae2f341f269245572cf3dae639f0636d6952f921012a7d0393"} Jul 10 16:51:48.342568 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:48.342521 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-96d6c78b5-2tdp8" event={"ID":"c64833da-5ba0-4a94-a050-88c6907421fa","Type":"ContainerStarted","Data":"f84a040a7d3d8e68d396e313f8cbeefcef30500061ba73b4a17117328d352538"} Jul 10 16:51:48.427297 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:48.427230 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-96d6c78b5-2tdp8" podStartSLOduration=1.938745823 podStartE2EDuration="2.427205965s" totalImagesPullingTime="488.460142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:51:48.37295972 +0000 UTC m=+927.794836188" watchObservedRunningTime="2026-07-10 16:51:48.427205965 +0000 UTC m=+927.849082435" Jul 10 16:51:48.429158 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:48.429130 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-845f7d5bd-4jch6"] Jul 10 16:51:48.429365 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:48.429339 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-845f7d5bd-4jch6" podUID="467b0c34-6f79-4ea9-8bce-86b53578b73f" containerName="authorino" containerID="cri-o://02221950de729602e0945bef7b35ba6610f3f0b04496ff36296580e6dbc2eaf9" gracePeriod=30 Jul 10 16:51:48.706730 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:48.706701 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-845f7d5bd-4jch6" Jul 10 16:51:48.831057 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:48.831022 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phwd2\" (UniqueName: \"kubernetes.io/projected/467b0c34-6f79-4ea9-8bce-86b53578b73f-kube-api-access-phwd2\") pod \"467b0c34-6f79-4ea9-8bce-86b53578b73f\" (UID: \"467b0c34-6f79-4ea9-8bce-86b53578b73f\") " Jul 10 16:51:48.831257 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:48.831123 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/467b0c34-6f79-4ea9-8bce-86b53578b73f-tls-cert\") pod \"467b0c34-6f79-4ea9-8bce-86b53578b73f\" (UID: \"467b0c34-6f79-4ea9-8bce-86b53578b73f\") " Jul 10 16:51:48.833348 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:48.833318 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467b0c34-6f79-4ea9-8bce-86b53578b73f-kube-api-access-phwd2" (OuterVolumeSpecName: "kube-api-access-phwd2") pod "467b0c34-6f79-4ea9-8bce-86b53578b73f" (UID: "467b0c34-6f79-4ea9-8bce-86b53578b73f"). InnerVolumeSpecName "kube-api-access-phwd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:51:48.844692 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:48.844659 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/467b0c34-6f79-4ea9-8bce-86b53578b73f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "467b0c34-6f79-4ea9-8bce-86b53578b73f" (UID: "467b0c34-6f79-4ea9-8bce-86b53578b73f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:51:48.932829 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:48.932706 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phwd2\" (UniqueName: \"kubernetes.io/projected/467b0c34-6f79-4ea9-8bce-86b53578b73f-kube-api-access-phwd2\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:51:48.932829 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:48.932776 2585 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/467b0c34-6f79-4ea9-8bce-86b53578b73f-tls-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 16:51:49.349383 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:49.349346 2585 generic.go:356] "Generic (PLEG): container finished" podID="467b0c34-6f79-4ea9-8bce-86b53578b73f" containerID="02221950de729602e0945bef7b35ba6610f3f0b04496ff36296580e6dbc2eaf9" exitCode=0 Jul 10 16:51:49.349862 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:49.349395 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-845f7d5bd-4jch6" Jul 10 16:51:49.349862 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:49.349436 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-845f7d5bd-4jch6" event={"ID":"467b0c34-6f79-4ea9-8bce-86b53578b73f","Type":"ContainerDied","Data":"02221950de729602e0945bef7b35ba6610f3f0b04496ff36296580e6dbc2eaf9"} Jul 10 16:51:49.349862 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:49.349481 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-845f7d5bd-4jch6" event={"ID":"467b0c34-6f79-4ea9-8bce-86b53578b73f","Type":"ContainerDied","Data":"9b28fa3d50d2d3c62467fa03a265e4bdb6397f050fb1b4f7234f64c3a602d0eb"} Jul 10 16:51:49.349862 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:49.349503 2585 scope.go:117] "RemoveContainer" containerID="02221950de729602e0945bef7b35ba6610f3f0b04496ff36296580e6dbc2eaf9" Jul 10 16:51:49.359320 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:49.359293 2585 scope.go:117] "RemoveContainer" containerID="02221950de729602e0945bef7b35ba6610f3f0b04496ff36296580e6dbc2eaf9" Jul 10 16:51:49.359730 ip-10-0-134-41 kubenswrapper[2585]: E0710 16:51:49.359703 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02221950de729602e0945bef7b35ba6610f3f0b04496ff36296580e6dbc2eaf9\": container with ID starting with 02221950de729602e0945bef7b35ba6610f3f0b04496ff36296580e6dbc2eaf9 not found: ID does not exist" containerID="02221950de729602e0945bef7b35ba6610f3f0b04496ff36296580e6dbc2eaf9" Jul 10 16:51:49.359836 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:49.359762 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02221950de729602e0945bef7b35ba6610f3f0b04496ff36296580e6dbc2eaf9"} err="failed to get container status \"02221950de729602e0945bef7b35ba6610f3f0b04496ff36296580e6dbc2eaf9\": rpc error: code = NotFound desc = could not find container \"02221950de729602e0945bef7b35ba6610f3f0b04496ff36296580e6dbc2eaf9\": container with ID starting with 02221950de729602e0945bef7b35ba6610f3f0b04496ff36296580e6dbc2eaf9 not found: ID does not exist" Jul 10 16:51:49.386362 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:49.386322 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-845f7d5bd-4jch6"] Jul 10 16:51:49.391266 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:49.391227 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-845f7d5bd-4jch6"] Jul 10 16:51:51.326535 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:51:51.326491 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="467b0c34-6f79-4ea9-8bce-86b53578b73f" path="/var/lib/kubelet/pods/467b0c34-6f79-4ea9-8bce-86b53578b73f/volumes" Jul 10 16:52:19.041422 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:52:19.041371 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:52:28.813886 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:52:28.813846 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:52:39.901315 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:52:39.901270 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:52:48.804175 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:52:48.804073 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:53:00.018188 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:53:00.018150 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:53:26.129876 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:53:26.129835 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:53:38.604921 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:53:38.604888 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:54:10.702241 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:54:10.702201 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:54:18.391500 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:54:18.391412 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:54:27.600354 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:54:27.600314 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:54:34.991606 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:54:34.991559 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:54:44.701444 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:54:44.701406 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:54:54.705129 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:54:54.705095 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:55:05.704486 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:55:05.704449 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:55:15.406270 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:55:15.406224 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:55:38.099733 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:55:38.099699 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:55:49.092705 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:55:49.092632 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:56:11.701503 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:56:11.701452 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:56:21.070669 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:56:21.070637 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 16:56:21.074824 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:56:21.074799 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 16:56:22.796043 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:56:22.795998 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:56:37.504329 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:56:37.504281 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:56:46.811595 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:56:46.811555 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:56:58.401588 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:56:58.401551 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:57:10.404498 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:57:10.404456 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:59:02.798141 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:59:02.798106 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:59:14.693639 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:59:14.693604 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:59:23.098231 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:59:23.098200 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:59:35.391790 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:59:35.391757 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:59:47.216255 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:59:47.216220 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 16:59:55.692445 ip-10-0-134-41 kubenswrapper[2585]: I0710 16:59:55.692408 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:00:03.997177 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:00:03.997136 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:00:13.892980 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:00:13.892899 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:00:22.397260 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:00:22.397218 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:00:32.297367 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:00:32.297329 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:00:44.409132 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:00:44.409099 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:00:52.111406 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:00:52.111364 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:01:09.311567 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:01:09.311525 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:01:18.613337 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:01:18.613306 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:01:21.106535 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:01:21.106507 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 17:01:21.114843 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:01:21.114821 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 17:01:28.909953 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:01:28.909913 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:01:39.110248 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:01:39.110214 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:01:48.509789 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:01:48.509684 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:02:01.301180 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:02:01.301146 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:02:12.991888 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:02:12.991856 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:02:25.489959 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:02:25.489929 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:02:35.893811 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:02:35.893771 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:02:45.096512 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:02:45.096472 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:02:54.293258 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:02:54.293224 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:03:08.703441 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:03:08.703400 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:03:18.295733 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:03:18.295635 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:03:27.619360 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:03:27.619326 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:03:37.391342 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:03:37.391307 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:03:49.795316 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:03:49.795284 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:03:57.799558 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:03:57.799522 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:04:09.801148 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:04:09.801113 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:04:21.309199 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:04:21.309161 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:04:34.399455 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:04:34.399415 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:04:43.596510 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:04:43.596422 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:04:54.101179 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:04:54.101143 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:05:06.317926 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:06.317889 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:05:12.352683 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.352645 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk"] Jul 10 17:05:12.353078 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.353023 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="467b0c34-6f79-4ea9-8bce-86b53578b73f" containerName="authorino" Jul 10 17:05:12.353078 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.353035 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="467b0c34-6f79-4ea9-8bce-86b53578b73f" containerName="authorino" Jul 10 17:05:12.353154 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.353112 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="467b0c34-6f79-4ea9-8bce-86b53578b73f" containerName="authorino" Jul 10 17:05:12.356119 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.356102 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.364939 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.364920 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7c0fd7dc-openshift-default-dockercfg-qz98c\"" Jul 10 17:05:12.403871 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.403826 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-envoy\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.404086 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.403883 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-data\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.404086 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.403920 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-workload-socket\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.404086 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.403967 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-workload-certs\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.404086 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.404028 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-podinfo\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.404287 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.404097 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp29w\" (UniqueName: \"kubernetes.io/projected/679bc91a-7b7f-446f-b562-9ad5528b116c-kube-api-access-bp29w\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.404372 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.404329 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-token\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.404576 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.404553 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-credential-socket\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.404677 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.404583 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/679bc91a-7b7f-446f-b562-9ad5528b116c-istiod-ca-cert\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.466470 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.466431 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk"] Jul 10 17:05:12.505414 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.505372 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-token\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.505414 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.505415 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-credential-socket\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.505683 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.505438 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/679bc91a-7b7f-446f-b562-9ad5528b116c-istiod-ca-cert\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.505683 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.505480 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-envoy\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.505683 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.505501 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-data\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.505683 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.505534 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-workload-socket\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.505683 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.505560 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-workload-certs\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.505961 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.505788 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-podinfo\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.505961 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.505846 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp29w\" (UniqueName: \"kubernetes.io/projected/679bc91a-7b7f-446f-b562-9ad5528b116c-kube-api-access-bp29w\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.505961 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.505910 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-credential-socket\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.505961 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.505935 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-workload-certs\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.506150 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.506044 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-data\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.506150 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.506096 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-workload-socket\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.506397 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.506376 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/679bc91a-7b7f-446f-b562-9ad5528b116c-istiod-ca-cert\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.507865 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.507835 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-envoy\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.508165 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.508147 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-podinfo\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.515872 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.515847 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-token\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.517259 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.517239 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp29w\" (UniqueName: \"kubernetes.io/projected/679bc91a-7b7f-446f-b562-9ad5528b116c-kube-api-access-bp29w\") pod \"e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.666328 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.666288 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:12.818780 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.818731 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk"] Jul 10 17:05:12.820282 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:05:12.820254 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679bc91a_7b7f_446f_b562_9ad5528b116c.slice/crio-9bedd39b8ca653f144e5c9d743534bb067b378251e2ba27705adf5724c680c37 WatchSource:0}: Error finding container 9bedd39b8ca653f144e5c9d743534bb067b378251e2ba27705adf5724c680c37: Status 404 returned error can't find the container with id 9bedd39b8ca653f144e5c9d743534bb067b378251e2ba27705adf5724c680c37 Jul 10 17:05:12.822257 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.822227 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:05:12.822345 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.822290 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:05:12.822345 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:12.822316 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:05:13.694405 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:13.694350 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" event={"ID":"679bc91a-7b7f-446f-b562-9ad5528b116c","Type":"ContainerStarted","Data":"7f1b0798ff20e5eb0ac72eb2ababfb2c6b5bfe69fa5d840b9218535980d68d69"} Jul 10 17:05:13.694839 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:13.694426 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" event={"ID":"679bc91a-7b7f-446f-b562-9ad5528b116c","Type":"ContainerStarted","Data":"9bedd39b8ca653f144e5c9d743534bb067b378251e2ba27705adf5724c680c37"} Jul 10 17:05:13.738921 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:13.738864 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" podStartSLOduration=1.7388402680000001 podStartE2EDuration="1.738840268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:05:13.738137729 +0000 UTC m=+1733.160014195" watchObservedRunningTime="2026-07-10 17:05:13.738840268 +0000 UTC m=+1733.160716729" Jul 10 17:05:14.667446 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:14.667408 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:14.672454 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:14.672429 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:14.700793 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:14.700759 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:14.701733 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:14.701713 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:17.326185 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:17.326152 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:05:25.766888 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:25.766802 2585 generic.go:356] "Generic (PLEG): container finished" podID="679bc91a-7b7f-446f-b562-9ad5528b116c" containerID="7f1b0798ff20e5eb0ac72eb2ababfb2c6b5bfe69fa5d840b9218535980d68d69" exitCode=0 Jul 10 17:05:25.767397 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:25.766880 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" event={"ID":"679bc91a-7b7f-446f-b562-9ad5528b116c","Type":"ContainerDied","Data":"7f1b0798ff20e5eb0ac72eb2ababfb2c6b5bfe69fa5d840b9218535980d68d69"} Jul 10 17:05:25.767476 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:25.767402 2585 scope.go:117] "RemoveContainer" containerID="7f1b0798ff20e5eb0ac72eb2ababfb2c6b5bfe69fa5d840b9218535980d68d69" Jul 10 17:05:25.768502 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:25.768471 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:05:25.768604 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:25.768543 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:05:25.768604 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:25.768573 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:05:26.774595 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:26.774562 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" event={"ID":"679bc91a-7b7f-446f-b562-9ad5528b116c","Type":"ContainerStarted","Data":"33a84c8f31fa3e7ba76fcc9d725d5858468e7fc6553725e3cbb89f269fbc330a"} Jul 10 17:05:27.667259 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:27.667223 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:27.668527 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:27.668500 2585 patch_prober.go:28] interesting pod/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 10 17:05:27.668596 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:27.668557 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" podUID="679bc91a-7b7f-446f-b562-9ad5528b116c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 10 17:05:28.667137 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:28.667098 2585 patch_prober.go:28] interesting pod/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 10 17:05:28.667537 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:28.667170 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" podUID="679bc91a-7b7f-446f-b562-9ad5528b116c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 10 17:05:29.666858 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:29.666824 2585 patch_prober.go:28] interesting pod/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 10 17:05:29.667045 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:29.666892 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" podUID="679bc91a-7b7f-446f-b562-9ad5528b116c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 10 17:05:29.701098 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:29.701067 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:30.666986 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:30.666946 2585 patch_prober.go:28] interesting pod/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 10 17:05:30.667172 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:30.667015 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" podUID="679bc91a-7b7f-446f-b562-9ad5528b116c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 10 17:05:31.670670 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:31.670641 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:05:31.671707 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:05:31.671687 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:06:21.143231 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:21.143201 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 17:06:21.152533 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:21.152507 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 17:06:31.890763 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:31.890716 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:06:43.805772 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:43.805726 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk"] Jul 10 17:06:43.806193 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:43.805937 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" podUID="679bc91a-7b7f-446f-b562-9ad5528b116c" containerName="istio-proxy" containerID="cri-o://33a84c8f31fa3e7ba76fcc9d725d5858468e7fc6553725e3cbb89f269fbc330a" gracePeriod=30 Jul 10 17:06:44.147204 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.146671 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp"] Jul 10 17:06:44.149801 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.149773 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.152704 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.152684 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fdbb8df2-openshift-default-dockercfg-7chqh\"" Jul 10 17:06:44.163464 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.163439 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp"] Jul 10 17:06:44.245033 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.245002 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlf7d\" (UniqueName: \"kubernetes.io/projected/0f75671a-c68c-42ed-951f-e80952417fdc-kube-api-access-jlf7d\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.245163 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.245042 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f75671a-c68c-42ed-951f-e80952417fdc-istiod-ca-cert\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.245163 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.245146 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-istio-envoy\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.245262 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.245182 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-workload-certs\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.245262 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.245205 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-istio-data\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.245262 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.245221 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-workload-socket\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.245262 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.245252 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-credential-socket\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.245476 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.245342 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f75671a-c68c-42ed-951f-e80952417fdc-istio-token\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.245476 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.245419 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f75671a-c68c-42ed-951f-e80952417fdc-istio-podinfo\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.346824 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.346789 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f75671a-c68c-42ed-951f-e80952417fdc-istio-token\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.347003 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.346873 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f75671a-c68c-42ed-951f-e80952417fdc-istio-podinfo\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.347003 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.346904 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlf7d\" (UniqueName: \"kubernetes.io/projected/0f75671a-c68c-42ed-951f-e80952417fdc-kube-api-access-jlf7d\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.347003 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.346935 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f75671a-c68c-42ed-951f-e80952417fdc-istiod-ca-cert\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.347003 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.346989 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-istio-envoy\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.347221 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.347020 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-workload-certs\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.347221 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.347055 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-istio-data\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.347221 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.347104 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-workload-socket\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.347221 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.347143 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-credential-socket\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.347456 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.347437 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-istio-data\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.347540 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.347500 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-workload-certs\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.347590 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.347563 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-credential-socket\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.347670 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.347639 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-workload-socket\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.348044 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.348014 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f75671a-c68c-42ed-951f-e80952417fdc-istiod-ca-cert\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.349407 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.349388 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-istio-envoy\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.349679 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.349661 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f75671a-c68c-42ed-951f-e80952417fdc-istio-podinfo\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.355411 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.355388 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f75671a-c68c-42ed-951f-e80952417fdc-istio-token\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.355642 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.355623 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlf7d\" (UniqueName: \"kubernetes.io/projected/0f75671a-c68c-42ed-951f-e80952417fdc-kube-api-access-jlf7d\") pod \"e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.463127 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.463064 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:44.604031 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.603997 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp"] Jul 10 17:06:44.605058 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:06:44.605030 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f75671a_c68c_42ed_951f_e80952417fdc.slice/crio-b457d1e45a34d60a6cddb6d6ea93f4bd435b11a9de52129b1966cc9a2af0319a WatchSource:0}: Error finding container b457d1e45a34d60a6cddb6d6ea93f4bd435b11a9de52129b1966cc9a2af0319a: Status 404 returned error can't find the container with id b457d1e45a34d60a6cddb6d6ea93f4bd435b11a9de52129b1966cc9a2af0319a Jul 10 17:06:44.607445 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.607401 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:06:44.607530 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.607495 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:06:44.607569 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:44.607542 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:06:45.195363 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:45.195329 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" event={"ID":"0f75671a-c68c-42ed-951f-e80952417fdc","Type":"ContainerStarted","Data":"72e595a779e4b7acc64fead27d3f170f8fb560854f2b5cccda0a6214671049f2"} Jul 10 17:06:45.195813 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:45.195408 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" event={"ID":"0f75671a-c68c-42ed-951f-e80952417fdc","Type":"ContainerStarted","Data":"b457d1e45a34d60a6cddb6d6ea93f4bd435b11a9de52129b1966cc9a2af0319a"} Jul 10 17:06:45.218965 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:45.218919 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" podStartSLOduration=1.218905124 podStartE2EDuration="1.218905124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:06:45.216703726 +0000 UTC m=+1824.638580221" watchObservedRunningTime="2026-07-10 17:06:45.218905124 +0000 UTC m=+1824.640781590" Jul 10 17:06:45.463269 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:45.463192 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:45.468012 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:45.467987 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:46.200998 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:46.200962 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:46.201891 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:46.201873 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:46.673846 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:46.673813 2585 patch_prober.go:28] interesting pod/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 17:06:46.674001 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:46.673866 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" podUID="679bc91a-7b7f-446f-b562-9ad5528b116c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 17:06:48.103398 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:48.103365 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:06:49.090839 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.090811 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:06:49.190613 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.190577 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-workload-certs\") pod \"679bc91a-7b7f-446f-b562-9ad5528b116c\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " Jul 10 17:06:49.191054 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.190697 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-podinfo\") pod \"679bc91a-7b7f-446f-b562-9ad5528b116c\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " Jul 10 17:06:49.191054 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.190735 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-workload-socket\") pod \"679bc91a-7b7f-446f-b562-9ad5528b116c\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " Jul 10 17:06:49.191054 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.190783 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-data\") pod \"679bc91a-7b7f-446f-b562-9ad5528b116c\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " Jul 10 17:06:49.191054 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.190883 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "679bc91a-7b7f-446f-b562-9ad5528b116c" (UID: "679bc91a-7b7f-446f-b562-9ad5528b116c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:06:49.191054 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.190945 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-credential-socket\") pod \"679bc91a-7b7f-446f-b562-9ad5528b116c\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " Jul 10 17:06:49.191054 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.190980 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp29w\" (UniqueName: \"kubernetes.io/projected/679bc91a-7b7f-446f-b562-9ad5528b116c-kube-api-access-bp29w\") pod \"679bc91a-7b7f-446f-b562-9ad5528b116c\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " Jul 10 17:06:49.191054 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.191027 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "679bc91a-7b7f-446f-b562-9ad5528b116c" (UID: "679bc91a-7b7f-446f-b562-9ad5528b116c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:06:49.191054 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.191049 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-token\") pod \"679bc91a-7b7f-446f-b562-9ad5528b116c\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " Jul 10 17:06:49.191431 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.191085 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-envoy\") pod \"679bc91a-7b7f-446f-b562-9ad5528b116c\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " Jul 10 17:06:49.191431 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.191113 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/679bc91a-7b7f-446f-b562-9ad5528b116c-istiod-ca-cert\") pod \"679bc91a-7b7f-446f-b562-9ad5528b116c\" (UID: \"679bc91a-7b7f-446f-b562-9ad5528b116c\") " Jul 10 17:06:49.191431 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.191198 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "679bc91a-7b7f-446f-b562-9ad5528b116c" (UID: "679bc91a-7b7f-446f-b562-9ad5528b116c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:06:49.191579 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.191445 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-workload-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:06:49.191579 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.191463 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-credential-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:06:49.191579 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.191478 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-workload-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:06:49.191838 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.191761 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-data" (OuterVolumeSpecName: "istio-data") pod "679bc91a-7b7f-446f-b562-9ad5528b116c" (UID: "679bc91a-7b7f-446f-b562-9ad5528b116c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:06:49.192011 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.191775 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679bc91a-7b7f-446f-b562-9ad5528b116c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "679bc91a-7b7f-446f-b562-9ad5528b116c" (UID: "679bc91a-7b7f-446f-b562-9ad5528b116c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:06:49.193589 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.193565 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "679bc91a-7b7f-446f-b562-9ad5528b116c" (UID: "679bc91a-7b7f-446f-b562-9ad5528b116c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:06:49.193857 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.193835 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "679bc91a-7b7f-446f-b562-9ad5528b116c" (UID: "679bc91a-7b7f-446f-b562-9ad5528b116c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:06:49.193941 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.193921 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-token" (OuterVolumeSpecName: "istio-token") pod "679bc91a-7b7f-446f-b562-9ad5528b116c" (UID: "679bc91a-7b7f-446f-b562-9ad5528b116c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:06:49.193999 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.193943 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679bc91a-7b7f-446f-b562-9ad5528b116c-kube-api-access-bp29w" (OuterVolumeSpecName: "kube-api-access-bp29w") pod "679bc91a-7b7f-446f-b562-9ad5528b116c" (UID: "679bc91a-7b7f-446f-b562-9ad5528b116c"). InnerVolumeSpecName "kube-api-access-bp29w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:06:49.220441 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.220416 2585 generic.go:356] "Generic (PLEG): container finished" podID="679bc91a-7b7f-446f-b562-9ad5528b116c" containerID="33a84c8f31fa3e7ba76fcc9d725d5858468e7fc6553725e3cbb89f269fbc330a" exitCode=0 Jul 10 17:06:49.220531 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.220464 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" event={"ID":"679bc91a-7b7f-446f-b562-9ad5528b116c","Type":"ContainerDied","Data":"33a84c8f31fa3e7ba76fcc9d725d5858468e7fc6553725e3cbb89f269fbc330a"} Jul 10 17:06:49.220571 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.220532 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" event={"ID":"679bc91a-7b7f-446f-b562-9ad5528b116c","Type":"ContainerDied","Data":"9bedd39b8ca653f144e5c9d743534bb067b378251e2ba27705adf5724c680c37"} Jul 10 17:06:49.220571 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.220470 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk" Jul 10 17:06:49.220571 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.220556 2585 scope.go:117] "RemoveContainer" containerID="33a84c8f31fa3e7ba76fcc9d725d5858468e7fc6553725e3cbb89f269fbc330a" Jul 10 17:06:49.230420 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.230405 2585 scope.go:117] "RemoveContainer" containerID="7f1b0798ff20e5eb0ac72eb2ababfb2c6b5bfe69fa5d840b9218535980d68d69" Jul 10 17:06:49.263711 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.263689 2585 scope.go:117] "RemoveContainer" containerID="33a84c8f31fa3e7ba76fcc9d725d5858468e7fc6553725e3cbb89f269fbc330a" Jul 10 17:06:49.264085 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:06:49.264064 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a84c8f31fa3e7ba76fcc9d725d5858468e7fc6553725e3cbb89f269fbc330a\": container with ID starting with 33a84c8f31fa3e7ba76fcc9d725d5858468e7fc6553725e3cbb89f269fbc330a not found: ID does not exist" containerID="33a84c8f31fa3e7ba76fcc9d725d5858468e7fc6553725e3cbb89f269fbc330a" Jul 10 17:06:49.264194 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.264105 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a84c8f31fa3e7ba76fcc9d725d5858468e7fc6553725e3cbb89f269fbc330a"} err="failed to get container status \"33a84c8f31fa3e7ba76fcc9d725d5858468e7fc6553725e3cbb89f269fbc330a\": rpc error: code = NotFound desc = could not find container \"33a84c8f31fa3e7ba76fcc9d725d5858468e7fc6553725e3cbb89f269fbc330a\": container with ID starting with 33a84c8f31fa3e7ba76fcc9d725d5858468e7fc6553725e3cbb89f269fbc330a not found: ID does not exist" Jul 10 17:06:49.264194 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.264126 2585 scope.go:117] "RemoveContainer" containerID="7f1b0798ff20e5eb0ac72eb2ababfb2c6b5bfe69fa5d840b9218535980d68d69" Jul 10 17:06:49.264476 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:06:49.264456 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f1b0798ff20e5eb0ac72eb2ababfb2c6b5bfe69fa5d840b9218535980d68d69\": container with ID starting with 7f1b0798ff20e5eb0ac72eb2ababfb2c6b5bfe69fa5d840b9218535980d68d69 not found: ID does not exist" containerID="7f1b0798ff20e5eb0ac72eb2ababfb2c6b5bfe69fa5d840b9218535980d68d69" Jul 10 17:06:49.264539 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.264483 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f1b0798ff20e5eb0ac72eb2ababfb2c6b5bfe69fa5d840b9218535980d68d69"} err="failed to get container status \"7f1b0798ff20e5eb0ac72eb2ababfb2c6b5bfe69fa5d840b9218535980d68d69\": rpc error: code = NotFound desc = could not find container \"7f1b0798ff20e5eb0ac72eb2ababfb2c6b5bfe69fa5d840b9218535980d68d69\": container with ID starting with 7f1b0798ff20e5eb0ac72eb2ababfb2c6b5bfe69fa5d840b9218535980d68d69 not found: ID does not exist" Jul 10 17:06:49.264945 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.264925 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk"] Jul 10 17:06:49.269098 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.269078 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7c0fd7dc-openshift-default-b6d99db8c-k5wjk"] Jul 10 17:06:49.292108 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.292081 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-token\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:06:49.292108 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.292107 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-envoy\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:06:49.292246 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.292116 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/679bc91a-7b7f-446f-b562-9ad5528b116c-istiod-ca-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:06:49.292246 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.292127 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-podinfo\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:06:49.292246 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.292153 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/679bc91a-7b7f-446f-b562-9ad5528b116c-istio-data\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:06:49.292246 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.292161 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp29w\" (UniqueName: \"kubernetes.io/projected/679bc91a-7b7f-446f-b562-9ad5528b116c-kube-api-access-bp29w\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:06:49.326454 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:49.326399 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="679bc91a-7b7f-446f-b562-9ad5528b116c" path="/var/lib/kubelet/pods/679bc91a-7b7f-446f-b562-9ad5528b116c/volumes" Jul 10 17:06:56.263011 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:56.262910 2585 generic.go:356] "Generic (PLEG): container finished" podID="0f75671a-c68c-42ed-951f-e80952417fdc" containerID="72e595a779e4b7acc64fead27d3f170f8fb560854f2b5cccda0a6214671049f2" exitCode=0 Jul 10 17:06:56.263011 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:56.262985 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" event={"ID":"0f75671a-c68c-42ed-951f-e80952417fdc","Type":"ContainerDied","Data":"72e595a779e4b7acc64fead27d3f170f8fb560854f2b5cccda0a6214671049f2"} Jul 10 17:06:56.263429 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:56.263347 2585 scope.go:117] "RemoveContainer" containerID="72e595a779e4b7acc64fead27d3f170f8fb560854f2b5cccda0a6214671049f2" Jul 10 17:06:56.264395 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:56.264369 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:06:56.264450 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:56.264436 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:06:56.264499 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:56.264487 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:06:57.273055 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:57.273019 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" event={"ID":"0f75671a-c68c-42ed-951f-e80952417fdc","Type":"ContainerStarted","Data":"5b8d4937373050087201b40ec2eb7d3bf6e978630d3d9192363504888dc7306c"} Jul 10 17:06:57.463648 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:57.463612 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:06:57.464929 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:57.464896 2585 patch_prober.go:28] interesting pod/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 10 17:06:57.465062 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:57.464943 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" podUID="0f75671a-c68c-42ed-951f-e80952417fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 10 17:06:58.464250 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:58.464217 2585 patch_prober.go:28] interesting pod/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 10 17:06:58.464629 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:58.464273 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" podUID="0f75671a-c68c-42ed-951f-e80952417fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 10 17:06:59.463723 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:59.463686 2585 patch_prober.go:28] interesting pod/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 10 17:06:59.463936 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:06:59.463782 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" podUID="0f75671a-c68c-42ed-951f-e80952417fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 10 17:07:00.463688 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:07:00.463650 2585 patch_prober.go:28] interesting pod/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 10 17:07:00.464078 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:07:00.463701 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" podUID="0f75671a-c68c-42ed-951f-e80952417fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 10 17:07:01.200915 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:07:01.200877 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:07:01.464156 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:07:01.464034 2585 patch_prober.go:28] interesting pod/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 10 17:07:01.464156 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:07:01.464106 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" podUID="0f75671a-c68c-42ed-951f-e80952417fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 10 17:07:02.467969 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:07:02.467938 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:07:02.468909 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:07:02.468889 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:08:03.794664 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:03.794627 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:08:16.462479 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.462442 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp"] Jul 10 17:08:16.462917 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.462653 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" podUID="0f75671a-c68c-42ed-951f-e80952417fdc" containerName="istio-proxy" containerID="cri-o://5b8d4937373050087201b40ec2eb7d3bf6e978630d3d9192363504888dc7306c" gracePeriod=30 Jul 10 17:08:16.768415 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.768382 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w"] Jul 10 17:08:16.768863 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.768846 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="679bc91a-7b7f-446f-b562-9ad5528b116c" containerName="istio-proxy" Jul 10 17:08:16.768863 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.768864 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="679bc91a-7b7f-446f-b562-9ad5528b116c" containerName="istio-proxy" Jul 10 17:08:16.768958 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.768876 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="679bc91a-7b7f-446f-b562-9ad5528b116c" containerName="istio-proxy" Jul 10 17:08:16.768958 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.768882 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="679bc91a-7b7f-446f-b562-9ad5528b116c" containerName="istio-proxy" Jul 10 17:08:16.768958 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.768945 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="679bc91a-7b7f-446f-b562-9ad5528b116c" containerName="istio-proxy" Jul 10 17:08:16.768958 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.768956 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="679bc91a-7b7f-446f-b562-9ad5528b116c" containerName="istio-proxy" Jul 10 17:08:16.771027 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.771012 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.773919 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.773900 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ef0ffa22-openshift-default-dockercfg-rd2pm\"" Jul 10 17:08:16.788716 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.788690 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w"] Jul 10 17:08:16.793472 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.793446 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-workload-socket\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.793606 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.793510 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-token\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.793606 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.793569 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-envoy\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.793606 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.793585 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-podinfo\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.793606 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.793602 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istiod-ca-cert\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.793795 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.793620 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-data\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.793795 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.793640 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-workload-certs\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.793795 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.793696 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmvqp\" (UniqueName: \"kubernetes.io/projected/ea7c43bc-3db9-4209-96a7-54ed8a38b641-kube-api-access-dmvqp\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.793795 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.793729 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-credential-socket\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.894549 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.894504 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-data\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.894549 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.894552 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-workload-certs\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.894872 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.894572 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmvqp\" (UniqueName: \"kubernetes.io/projected/ea7c43bc-3db9-4209-96a7-54ed8a38b641-kube-api-access-dmvqp\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.894872 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.894592 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-credential-socket\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.894872 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.894629 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-workload-socket\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.894872 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.894684 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-token\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.894872 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.894732 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-envoy\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.894872 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.894766 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-podinfo\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.894872 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.894790 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istiod-ca-cert\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.895174 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.895029 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-data\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.895174 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.895055 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-credential-socket\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.895174 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.895100 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-workload-certs\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.895174 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.895124 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-workload-socket\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.895559 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.895533 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istiod-ca-cert\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.897203 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.897178 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-envoy\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.897307 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.897287 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-podinfo\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.909012 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.908981 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmvqp\" (UniqueName: \"kubernetes.io/projected/ea7c43bc-3db9-4209-96a7-54ed8a38b641-kube-api-access-dmvqp\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:16.910094 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:16.910075 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-token\") pod \"e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:17.083784 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:17.083650 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:17.224062 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:17.224032 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w"] Jul 10 17:08:17.225649 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:08:17.225618 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7c43bc_3db9_4209_96a7_54ed8a38b641.slice/crio-db689da5467eac4a1e8e18e7dd42f72a7fedeb94b02d20109686d1349b277528 WatchSource:0}: Error finding container db689da5467eac4a1e8e18e7dd42f72a7fedeb94b02d20109686d1349b277528: Status 404 returned error can't find the container with id db689da5467eac4a1e8e18e7dd42f72a7fedeb94b02d20109686d1349b277528 Jul 10 17:08:17.227614 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:17.227582 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:08:17.227700 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:17.227649 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:08:17.227700 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:17.227678 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:08:17.470786 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:17.470722 2585 patch_prober.go:28] interesting pod/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 17:08:17.471254 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:17.470822 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" podUID="0f75671a-c68c-42ed-951f-e80952417fdc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 17:08:17.712825 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:17.712790 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" event={"ID":"ea7c43bc-3db9-4209-96a7-54ed8a38b641","Type":"ContainerStarted","Data":"b52451aa92678166e0a34f4c8a7272243f2f55449a4eb2c657509a079212f01f"} Jul 10 17:08:17.713038 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:17.712846 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" event={"ID":"ea7c43bc-3db9-4209-96a7-54ed8a38b641","Type":"ContainerStarted","Data":"db689da5467eac4a1e8e18e7dd42f72a7fedeb94b02d20109686d1349b277528"} Jul 10 17:08:17.736387 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:17.736255 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" podStartSLOduration=1.73623419 podStartE2EDuration="1.73623419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:08:17.735897683 +0000 UTC m=+1917.157774173" watchObservedRunningTime="2026-07-10 17:08:17.73623419 +0000 UTC m=+1917.158110658" Jul 10 17:08:18.084176 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:18.084066 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:18.089047 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:18.089022 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:18.718107 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:18.718071 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:18.719299 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:18.719276 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:20.500814 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:20.500766 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:08:21.762303 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.762264 2585 generic.go:356] "Generic (PLEG): container finished" podID="0f75671a-c68c-42ed-951f-e80952417fdc" containerID="5b8d4937373050087201b40ec2eb7d3bf6e978630d3d9192363504888dc7306c" exitCode=0 Jul 10 17:08:21.762661 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.762443 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" event={"ID":"0f75671a-c68c-42ed-951f-e80952417fdc","Type":"ContainerDied","Data":"5b8d4937373050087201b40ec2eb7d3bf6e978630d3d9192363504888dc7306c"} Jul 10 17:08:21.762661 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.762496 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" event={"ID":"0f75671a-c68c-42ed-951f-e80952417fdc","Type":"ContainerDied","Data":"b457d1e45a34d60a6cddb6d6ea93f4bd435b11a9de52129b1966cc9a2af0319a"} Jul 10 17:08:21.762661 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.762521 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b457d1e45a34d60a6cddb6d6ea93f4bd435b11a9de52129b1966cc9a2af0319a" Jul 10 17:08:21.762661 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.762540 2585 scope.go:117] "RemoveContainer" containerID="72e595a779e4b7acc64fead27d3f170f8fb560854f2b5cccda0a6214671049f2" Jul 10 17:08:21.769073 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.769049 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:08:21.852297 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.852251 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-workload-socket\") pod \"0f75671a-c68c-42ed-951f-e80952417fdc\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " Jul 10 17:08:21.852675 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.852565 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlf7d\" (UniqueName: \"kubernetes.io/projected/0f75671a-c68c-42ed-951f-e80952417fdc-kube-api-access-jlf7d\") pod \"0f75671a-c68c-42ed-951f-e80952417fdc\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " Jul 10 17:08:21.852675 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.852634 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-credential-socket\") pod \"0f75671a-c68c-42ed-951f-e80952417fdc\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " Jul 10 17:08:21.852950 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.852916 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-workload-certs\") pod \"0f75671a-c68c-42ed-951f-e80952417fdc\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " Jul 10 17:08:21.853134 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.853093 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f75671a-c68c-42ed-951f-e80952417fdc-istio-token\") pod \"0f75671a-c68c-42ed-951f-e80952417fdc\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " Jul 10 17:08:21.854018 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.853976 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0f75671a-c68c-42ed-951f-e80952417fdc" (UID: "0f75671a-c68c-42ed-951f-e80952417fdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:08:21.854508 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.854459 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0f75671a-c68c-42ed-951f-e80952417fdc" (UID: "0f75671a-c68c-42ed-951f-e80952417fdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:08:21.855315 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.855279 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f75671a-c68c-42ed-951f-e80952417fdc-istio-podinfo\") pod \"0f75671a-c68c-42ed-951f-e80952417fdc\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " Jul 10 17:08:21.859003 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.858920 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f75671a-c68c-42ed-951f-e80952417fdc-istiod-ca-cert\") pod \"0f75671a-c68c-42ed-951f-e80952417fdc\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " Jul 10 17:08:21.859003 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.858955 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-istio-envoy\") pod \"0f75671a-c68c-42ed-951f-e80952417fdc\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " Jul 10 17:08:21.859273 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.859231 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0f75671a-c68c-42ed-951f-e80952417fdc" (UID: "0f75671a-c68c-42ed-951f-e80952417fdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:08:21.861766 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.859469 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f75671a-c68c-42ed-951f-e80952417fdc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0f75671a-c68c-42ed-951f-e80952417fdc" (UID: "0f75671a-c68c-42ed-951f-e80952417fdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:08:21.862373 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.861892 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-istio-data\") pod \"0f75671a-c68c-42ed-951f-e80952417fdc\" (UID: \"0f75671a-c68c-42ed-951f-e80952417fdc\") " Jul 10 17:08:21.862373 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.862333 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f75671a-c68c-42ed-951f-e80952417fdc-istiod-ca-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:08:21.862771 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.862557 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-workload-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:08:21.862771 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.862581 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-credential-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:08:21.862771 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.862605 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-workload-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:08:21.863868 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.863847 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-istio-data" (OuterVolumeSpecName: "istio-data") pod "0f75671a-c68c-42ed-951f-e80952417fdc" (UID: "0f75671a-c68c-42ed-951f-e80952417fdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:08:21.866623 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.866561 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0f75671a-c68c-42ed-951f-e80952417fdc" (UID: "0f75671a-c68c-42ed-951f-e80952417fdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:08:21.866953 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.866899 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f75671a-c68c-42ed-951f-e80952417fdc-istio-token" (OuterVolumeSpecName: "istio-token") pod "0f75671a-c68c-42ed-951f-e80952417fdc" (UID: "0f75671a-c68c-42ed-951f-e80952417fdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:08:21.868433 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.868390 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f75671a-c68c-42ed-951f-e80952417fdc-kube-api-access-jlf7d" (OuterVolumeSpecName: "kube-api-access-jlf7d") pod "0f75671a-c68c-42ed-951f-e80952417fdc" (UID: "0f75671a-c68c-42ed-951f-e80952417fdc"). InnerVolumeSpecName "kube-api-access-jlf7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:08:21.869425 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.869390 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f75671a-c68c-42ed-951f-e80952417fdc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0f75671a-c68c-42ed-951f-e80952417fdc" (UID: "0f75671a-c68c-42ed-951f-e80952417fdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:08:21.963331 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.963295 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f75671a-c68c-42ed-951f-e80952417fdc-istio-token\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:08:21.963331 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.963324 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f75671a-c68c-42ed-951f-e80952417fdc-istio-podinfo\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:08:21.963331 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.963334 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-istio-envoy\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:08:21.963331 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.963342 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f75671a-c68c-42ed-951f-e80952417fdc-istio-data\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:08:21.963612 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:21.963350 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlf7d\" (UniqueName: \"kubernetes.io/projected/0f75671a-c68c-42ed-951f-e80952417fdc-kube-api-access-jlf7d\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:08:22.769219 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:22.769187 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp" Jul 10 17:08:22.806665 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:22.806624 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp"] Jul 10 17:08:22.811754 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:22.811717 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fdbb8df2-openshift-default-d6db66c9f-rjlmp"] Jul 10 17:08:23.326023 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:23.325942 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f75671a-c68c-42ed-951f-e80952417fdc" path="/var/lib/kubelet/pods/0f75671a-c68c-42ed-951f-e80952417fdc/volumes" Jul 10 17:08:28.806295 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:28.806198 2585 generic.go:356] "Generic (PLEG): container finished" podID="ea7c43bc-3db9-4209-96a7-54ed8a38b641" containerID="b52451aa92678166e0a34f4c8a7272243f2f55449a4eb2c657509a079212f01f" exitCode=0 Jul 10 17:08:28.806295 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:28.806268 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" event={"ID":"ea7c43bc-3db9-4209-96a7-54ed8a38b641","Type":"ContainerDied","Data":"b52451aa92678166e0a34f4c8a7272243f2f55449a4eb2c657509a079212f01f"} Jul 10 17:08:28.806738 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:28.806688 2585 scope.go:117] "RemoveContainer" containerID="b52451aa92678166e0a34f4c8a7272243f2f55449a4eb2c657509a079212f01f" Jul 10 17:08:28.807706 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:28.807673 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:08:28.807838 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:28.807774 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:08:28.807838 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:28.807826 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:08:29.813631 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:29.813593 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" event={"ID":"ea7c43bc-3db9-4209-96a7-54ed8a38b641","Type":"ContainerStarted","Data":"35e20556c80dabbb457b4d664e2fbf3fdf3daf1369a3181bc8185fb2edde5c2b"} Jul 10 17:08:30.084151 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:30.084045 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:30.085211 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:30.085185 2585 patch_prober.go:28] interesting pod/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 10 17:08:30.085277 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:30.085238 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" podUID="ea7c43bc-3db9-4209-96a7-54ed8a38b641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 10 17:08:31.084440 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:31.084409 2585 patch_prober.go:28] interesting pod/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 10 17:08:31.084860 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:31.084460 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" podUID="ea7c43bc-3db9-4209-96a7-54ed8a38b641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 10 17:08:32.084455 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:32.084416 2585 patch_prober.go:28] interesting pod/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 10 17:08:32.084963 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:32.084487 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" podUID="ea7c43bc-3db9-4209-96a7-54ed8a38b641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 10 17:08:33.084993 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:33.084952 2585 patch_prober.go:28] interesting pod/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 10 17:08:33.085412 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:33.085015 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" podUID="ea7c43bc-3db9-4209-96a7-54ed8a38b641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 10 17:08:33.718968 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:33.718922 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:34.088291 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:34.088212 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:08:34.089218 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:08:34.089197 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:09:25.705273 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:25.705234 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:09:37.645222 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:37.645186 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w"] Jul 10 17:09:37.645670 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:37.645416 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" podUID="ea7c43bc-3db9-4209-96a7-54ed8a38b641" containerName="istio-proxy" containerID="cri-o://35e20556c80dabbb457b4d664e2fbf3fdf3daf1369a3181bc8185fb2edde5c2b" gracePeriod=30 Jul 10 17:09:42.904123 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.904096 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:09:42.946614 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.946518 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-envoy\") pod \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " Jul 10 17:09:42.946614 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.946561 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-credential-socket\") pod \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " Jul 10 17:09:42.946614 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.946595 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istiod-ca-cert\") pod \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " Jul 10 17:09:42.946947 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.946632 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-podinfo\") pod \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " Jul 10 17:09:42.946947 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.946700 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmvqp\" (UniqueName: \"kubernetes.io/projected/ea7c43bc-3db9-4209-96a7-54ed8a38b641-kube-api-access-dmvqp\") pod \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " Jul 10 17:09:42.946947 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.946726 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-workload-socket\") pod \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " Jul 10 17:09:42.946947 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.946781 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-data\") pod \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " Jul 10 17:09:42.946947 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.946834 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-token\") pod \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " Jul 10 17:09:42.946947 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.946858 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-workload-certs\") pod \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\" (UID: \"ea7c43bc-3db9-4209-96a7-54ed8a38b641\") " Jul 10 17:09:42.946947 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.946921 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ea7c43bc-3db9-4209-96a7-54ed8a38b641" (UID: "ea7c43bc-3db9-4209-96a7-54ed8a38b641"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:09:42.947330 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.947207 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-credential-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:09:42.947609 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.947440 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ea7c43bc-3db9-4209-96a7-54ed8a38b641" (UID: "ea7c43bc-3db9-4209-96a7-54ed8a38b641"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:09:42.947922 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.947884 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ea7c43bc-3db9-4209-96a7-54ed8a38b641" (UID: "ea7c43bc-3db9-4209-96a7-54ed8a38b641"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:09:42.948144 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.948024 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ea7c43bc-3db9-4209-96a7-54ed8a38b641" (UID: "ea7c43bc-3db9-4209-96a7-54ed8a38b641"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:09:42.948279 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.948244 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-data" (OuterVolumeSpecName: "istio-data") pod "ea7c43bc-3db9-4209-96a7-54ed8a38b641" (UID: "ea7c43bc-3db9-4209-96a7-54ed8a38b641"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:09:42.949313 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.949291 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ea7c43bc-3db9-4209-96a7-54ed8a38b641" (UID: "ea7c43bc-3db9-4209-96a7-54ed8a38b641"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:09:42.950384 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.950341 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-token" (OuterVolumeSpecName: "istio-token") pod "ea7c43bc-3db9-4209-96a7-54ed8a38b641" (UID: "ea7c43bc-3db9-4209-96a7-54ed8a38b641"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:09:42.950487 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.950416 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ea7c43bc-3db9-4209-96a7-54ed8a38b641" (UID: "ea7c43bc-3db9-4209-96a7-54ed8a38b641"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:09:42.951878 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:42.951852 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7c43bc-3db9-4209-96a7-54ed8a38b641-kube-api-access-dmvqp" (OuterVolumeSpecName: "kube-api-access-dmvqp") pod "ea7c43bc-3db9-4209-96a7-54ed8a38b641" (UID: "ea7c43bc-3db9-4209-96a7-54ed8a38b641"). InnerVolumeSpecName "kube-api-access-dmvqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:09:43.047933 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.047893 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmvqp\" (UniqueName: \"kubernetes.io/projected/ea7c43bc-3db9-4209-96a7-54ed8a38b641-kube-api-access-dmvqp\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:09:43.047933 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.047927 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-workload-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:09:43.047933 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.047937 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-data\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:09:43.048253 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.047946 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-token\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:09:43.048253 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.047954 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-workload-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:09:43.048253 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.047962 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-envoy\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:09:43.048253 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.047969 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istiod-ca-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:09:43.048253 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.047977 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7c43bc-3db9-4209-96a7-54ed8a38b641-istio-podinfo\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:09:43.223968 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.223868 2585 generic.go:356] "Generic (PLEG): container finished" podID="ea7c43bc-3db9-4209-96a7-54ed8a38b641" containerID="35e20556c80dabbb457b4d664e2fbf3fdf3daf1369a3181bc8185fb2edde5c2b" exitCode=0 Jul 10 17:09:43.223968 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.223920 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" Jul 10 17:09:43.223968 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.223946 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" event={"ID":"ea7c43bc-3db9-4209-96a7-54ed8a38b641","Type":"ContainerDied","Data":"35e20556c80dabbb457b4d664e2fbf3fdf3daf1369a3181bc8185fb2edde5c2b"} Jul 10 17:09:43.224275 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.224010 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w" event={"ID":"ea7c43bc-3db9-4209-96a7-54ed8a38b641","Type":"ContainerDied","Data":"db689da5467eac4a1e8e18e7dd42f72a7fedeb94b02d20109686d1349b277528"} Jul 10 17:09:43.224275 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.224041 2585 scope.go:117] "RemoveContainer" containerID="35e20556c80dabbb457b4d664e2fbf3fdf3daf1369a3181bc8185fb2edde5c2b" Jul 10 17:09:43.237270 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.237070 2585 scope.go:117] "RemoveContainer" containerID="b52451aa92678166e0a34f4c8a7272243f2f55449a4eb2c657509a079212f01f" Jul 10 17:09:43.246577 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.246554 2585 scope.go:117] "RemoveContainer" containerID="35e20556c80dabbb457b4d664e2fbf3fdf3daf1369a3181bc8185fb2edde5c2b" Jul 10 17:09:43.246940 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:09:43.246915 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e20556c80dabbb457b4d664e2fbf3fdf3daf1369a3181bc8185fb2edde5c2b\": container with ID starting with 35e20556c80dabbb457b4d664e2fbf3fdf3daf1369a3181bc8185fb2edde5c2b not found: ID does not exist" containerID="35e20556c80dabbb457b4d664e2fbf3fdf3daf1369a3181bc8185fb2edde5c2b" Jul 10 17:09:43.247030 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.246951 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e20556c80dabbb457b4d664e2fbf3fdf3daf1369a3181bc8185fb2edde5c2b"} err="failed to get container status \"35e20556c80dabbb457b4d664e2fbf3fdf3daf1369a3181bc8185fb2edde5c2b\": rpc error: code = NotFound desc = could not find container \"35e20556c80dabbb457b4d664e2fbf3fdf3daf1369a3181bc8185fb2edde5c2b\": container with ID starting with 35e20556c80dabbb457b4d664e2fbf3fdf3daf1369a3181bc8185fb2edde5c2b not found: ID does not exist" Jul 10 17:09:43.247030 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.246970 2585 scope.go:117] "RemoveContainer" containerID="b52451aa92678166e0a34f4c8a7272243f2f55449a4eb2c657509a079212f01f" Jul 10 17:09:43.247255 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:09:43.247239 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52451aa92678166e0a34f4c8a7272243f2f55449a4eb2c657509a079212f01f\": container with ID starting with b52451aa92678166e0a34f4c8a7272243f2f55449a4eb2c657509a079212f01f not found: ID does not exist" containerID="b52451aa92678166e0a34f4c8a7272243f2f55449a4eb2c657509a079212f01f" Jul 10 17:09:43.247307 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.247258 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52451aa92678166e0a34f4c8a7272243f2f55449a4eb2c657509a079212f01f"} err="failed to get container status \"b52451aa92678166e0a34f4c8a7272243f2f55449a4eb2c657509a079212f01f\": rpc error: code = NotFound desc = could not find container \"b52451aa92678166e0a34f4c8a7272243f2f55449a4eb2c657509a079212f01f\": container with ID starting with b52451aa92678166e0a34f4c8a7272243f2f55449a4eb2c657509a079212f01f not found: ID does not exist" Jul 10 17:09:43.257621 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.257590 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w"] Jul 10 17:09:43.263572 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.263547 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ef0ffa22-openshift-default-8498b44675-kq28w"] Jul 10 17:09:43.327634 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:09:43.327600 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea7c43bc-3db9-4209-96a7-54ed8a38b641" path="/var/lib/kubelet/pods/ea7c43bc-3db9-4209-96a7-54ed8a38b641/volumes" Jul 10 17:11:00.195944 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:00.195897 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:11:08.997315 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:08.997273 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:11:13.672547 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.672512 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw"] Jul 10 17:11:13.673094 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.673073 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea7c43bc-3db9-4209-96a7-54ed8a38b641" containerName="istio-proxy" Jul 10 17:11:13.673183 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.673096 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea7c43bc-3db9-4209-96a7-54ed8a38b641" containerName="istio-proxy" Jul 10 17:11:13.673183 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.673126 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f75671a-c68c-42ed-951f-e80952417fdc" containerName="istio-proxy" Jul 10 17:11:13.673183 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.673136 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f75671a-c68c-42ed-951f-e80952417fdc" containerName="istio-proxy" Jul 10 17:11:13.673183 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.673148 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea7c43bc-3db9-4209-96a7-54ed8a38b641" containerName="istio-proxy" Jul 10 17:11:13.673183 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.673158 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea7c43bc-3db9-4209-96a7-54ed8a38b641" containerName="istio-proxy" Jul 10 17:11:13.673183 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.673173 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f75671a-c68c-42ed-951f-e80952417fdc" containerName="istio-proxy" Jul 10 17:11:13.673183 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.673181 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f75671a-c68c-42ed-951f-e80952417fdc" containerName="istio-proxy" Jul 10 17:11:13.673536 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.673265 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea7c43bc-3db9-4209-96a7-54ed8a38b641" containerName="istio-proxy" Jul 10 17:11:13.673536 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.673282 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f75671a-c68c-42ed-951f-e80952417fdc" containerName="istio-proxy" Jul 10 17:11:13.673536 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.673292 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f75671a-c68c-42ed-951f-e80952417fdc" containerName="istio-proxy" Jul 10 17:11:13.675958 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.675938 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.679406 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.679381 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e08efa8c-openshift-default-dockercfg-lq2sz\"" Jul 10 17:11:13.698184 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.698160 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw"] Jul 10 17:11:13.793148 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.793108 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-istio-data\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.793353 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.793157 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-istio-envoy\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.793353 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.793189 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-workload-certs\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.793353 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.793217 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-workload-socket\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.793353 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.793242 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac7f002-d45b-415c-b77f-56a86c280f33-istio-token\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.793353 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.793336 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5mt6\" (UniqueName: \"kubernetes.io/projected/dac7f002-d45b-415c-b77f-56a86c280f33-kube-api-access-m5mt6\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.793525 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.793370 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac7f002-d45b-415c-b77f-56a86c280f33-istio-podinfo\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.793525 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.793431 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac7f002-d45b-415c-b77f-56a86c280f33-istiod-ca-cert\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.793525 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.793465 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-credential-socket\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.894776 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.894714 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-workload-certs\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.894963 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.894787 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-workload-socket\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.894963 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.894813 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac7f002-d45b-415c-b77f-56a86c280f33-istio-token\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.894963 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.894840 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5mt6\" (UniqueName: \"kubernetes.io/projected/dac7f002-d45b-415c-b77f-56a86c280f33-kube-api-access-m5mt6\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.894963 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.894883 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac7f002-d45b-415c-b77f-56a86c280f33-istio-podinfo\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.894963 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.894921 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac7f002-d45b-415c-b77f-56a86c280f33-istiod-ca-cert\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.894963 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.894957 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-credential-socket\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.895290 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.895043 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-istio-data\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.895290 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.895086 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-istio-envoy\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.895290 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.895143 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-workload-certs\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.895451 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.895363 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-credential-socket\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.895451 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.895416 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-istio-data\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.895552 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.895473 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-workload-socket\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.895623 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.895605 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac7f002-d45b-415c-b77f-56a86c280f33-istiod-ca-cert\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.897223 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.897201 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-istio-envoy\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.897490 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.897474 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac7f002-d45b-415c-b77f-56a86c280f33-istio-podinfo\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.904468 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.904436 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac7f002-d45b-415c-b77f-56a86c280f33-istio-token\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.906947 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.906925 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5mt6\" (UniqueName: \"kubernetes.io/projected/dac7f002-d45b-415c-b77f-56a86c280f33-kube-api-access-m5mt6\") pod \"e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:13.992377 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:13.992292 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:14.138539 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:14.138509 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw"] Jul 10 17:11:14.139824 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:11:14.139800 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac7f002_d45b_415c_b77f_56a86c280f33.slice/crio-d7aeca03d330ee84ec213a5875391d2e10251099ce07890a17b45594c4d7c3d2 WatchSource:0}: Error finding container d7aeca03d330ee84ec213a5875391d2e10251099ce07890a17b45594c4d7c3d2: Status 404 returned error can't find the container with id d7aeca03d330ee84ec213a5875391d2e10251099ce07890a17b45594c4d7c3d2 Jul 10 17:11:14.141935 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:14.141904 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:11:14.142009 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:14.141970 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:11:14.142009 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:14.142000 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:11:14.717710 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:14.717673 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" event={"ID":"dac7f002-d45b-415c-b77f-56a86c280f33","Type":"ContainerStarted","Data":"6178123ebaf252b6cbff6efca461a2a02b221867096f09551567ccf1dc759587"} Jul 10 17:11:14.718212 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:14.717729 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" event={"ID":"dac7f002-d45b-415c-b77f-56a86c280f33","Type":"ContainerStarted","Data":"d7aeca03d330ee84ec213a5875391d2e10251099ce07890a17b45594c4d7c3d2"} Jul 10 17:11:14.742107 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:14.742053 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" podStartSLOduration=1.742037925 podStartE2EDuration="1.742037925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:14.740347545 +0000 UTC m=+2094.162224042" watchObservedRunningTime="2026-07-10 17:11:14.742037925 +0000 UTC m=+2094.163914391" Jul 10 17:11:14.993163 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:14.993052 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:15.997924 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:15.997893 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:16.728887 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:16.728855 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:16.729982 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:16.729959 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:21.179299 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:21.179269 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 17:11:21.189738 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:21.189717 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 17:11:22.207083 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:22.207043 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:11:30.810553 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:30.810515 2585 generic.go:356] "Generic (PLEG): container finished" podID="dac7f002-d45b-415c-b77f-56a86c280f33" containerID="6178123ebaf252b6cbff6efca461a2a02b221867096f09551567ccf1dc759587" exitCode=0 Jul 10 17:11:30.810990 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:30.810588 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" event={"ID":"dac7f002-d45b-415c-b77f-56a86c280f33","Type":"ContainerDied","Data":"6178123ebaf252b6cbff6efca461a2a02b221867096f09551567ccf1dc759587"} Jul 10 17:11:30.811036 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:30.811020 2585 scope.go:117] "RemoveContainer" containerID="6178123ebaf252b6cbff6efca461a2a02b221867096f09551567ccf1dc759587" Jul 10 17:11:30.811970 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:30.811945 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:11:30.812031 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:30.812008 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:11:30.812070 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:30.812040 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:11:31.728821 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:31.728785 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:31.818498 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:31.818456 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" event={"ID":"dac7f002-d45b-415c-b77f-56a86c280f33","Type":"ContainerStarted","Data":"784d8590dabbbabad8aab35d2cf6571263b4795f89db79ac90855de8993b0134"} Jul 10 17:11:31.992885 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:31.992793 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:31.994173 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:31.994150 2585 patch_prober.go:28] interesting pod/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 10 17:11:31.994250 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:31.994231 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" podUID="dac7f002-d45b-415c-b77f-56a86c280f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 10 17:11:32.993503 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:32.993470 2585 patch_prober.go:28] interesting pod/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 10 17:11:32.993924 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:32.993535 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" podUID="dac7f002-d45b-415c-b77f-56a86c280f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 10 17:11:33.993374 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:33.993324 2585 patch_prober.go:28] interesting pod/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 10 17:11:33.993571 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:33.993412 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" podUID="dac7f002-d45b-415c-b77f-56a86c280f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 10 17:11:34.993140 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:34.993103 2585 patch_prober.go:28] interesting pod/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 10 17:11:34.993341 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:34.993179 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" podUID="dac7f002-d45b-415c-b77f-56a86c280f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 10 17:11:36.006425 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:36.006393 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:36.006818 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:36.006675 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:36.007402 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:36.007385 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:42.489492 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:42.489454 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:11:54.433998 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:54.433960 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw"] Jul 10 17:11:54.434431 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:54.434193 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" podUID="dac7f002-d45b-415c-b77f-56a86c280f33" containerName="istio-proxy" containerID="cri-o://784d8590dabbbabad8aab35d2cf6571263b4795f89db79ac90855de8993b0134" gracePeriod=30 Jul 10 17:11:59.693091 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.693062 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:59.845903 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.845803 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac7f002-d45b-415c-b77f-56a86c280f33-istio-token\") pod \"dac7f002-d45b-415c-b77f-56a86c280f33\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " Jul 10 17:11:59.845903 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.845849 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-workload-certs\") pod \"dac7f002-d45b-415c-b77f-56a86c280f33\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " Jul 10 17:11:59.845903 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.845881 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-istio-data\") pod \"dac7f002-d45b-415c-b77f-56a86c280f33\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " Jul 10 17:11:59.846210 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.846051 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-credential-socket\") pod \"dac7f002-d45b-415c-b77f-56a86c280f33\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " Jul 10 17:11:59.846210 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.846084 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-istio-envoy\") pod \"dac7f002-d45b-415c-b77f-56a86c280f33\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " Jul 10 17:11:59.846210 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.846141 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac7f002-d45b-415c-b77f-56a86c280f33-istio-podinfo\") pod \"dac7f002-d45b-415c-b77f-56a86c280f33\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " Jul 10 17:11:59.846210 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.846150 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dac7f002-d45b-415c-b77f-56a86c280f33" (UID: "dac7f002-d45b-415c-b77f-56a86c280f33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:11:59.846210 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.846179 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac7f002-d45b-415c-b77f-56a86c280f33-istiod-ca-cert\") pod \"dac7f002-d45b-415c-b77f-56a86c280f33\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " Jul 10 17:11:59.846210 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.846208 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-workload-socket\") pod \"dac7f002-d45b-415c-b77f-56a86c280f33\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " Jul 10 17:11:59.846529 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.846272 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5mt6\" (UniqueName: \"kubernetes.io/projected/dac7f002-d45b-415c-b77f-56a86c280f33-kube-api-access-m5mt6\") pod \"dac7f002-d45b-415c-b77f-56a86c280f33\" (UID: \"dac7f002-d45b-415c-b77f-56a86c280f33\") " Jul 10 17:11:59.846529 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.846305 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dac7f002-d45b-415c-b77f-56a86c280f33" (UID: "dac7f002-d45b-415c-b77f-56a86c280f33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:11:59.846529 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.846507 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dac7f002-d45b-415c-b77f-56a86c280f33" (UID: "dac7f002-d45b-415c-b77f-56a86c280f33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:11:59.846692 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.846581 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac7f002-d45b-415c-b77f-56a86c280f33-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dac7f002-d45b-415c-b77f-56a86c280f33" (UID: "dac7f002-d45b-415c-b77f-56a86c280f33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:11:59.846692 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.846676 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-workload-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:11:59.846844 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.846695 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-credential-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:11:59.846844 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.846712 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac7f002-d45b-415c-b77f-56a86c280f33-istiod-ca-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:11:59.846844 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.846727 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-workload-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:11:59.847001 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.846900 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-istio-data" (OuterVolumeSpecName: "istio-data") pod "dac7f002-d45b-415c-b77f-56a86c280f33" (UID: "dac7f002-d45b-415c-b77f-56a86c280f33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:11:59.848229 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.848198 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac7f002-d45b-415c-b77f-56a86c280f33-istio-token" (OuterVolumeSpecName: "istio-token") pod "dac7f002-d45b-415c-b77f-56a86c280f33" (UID: "dac7f002-d45b-415c-b77f-56a86c280f33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:11:59.848229 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.848213 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dac7f002-d45b-415c-b77f-56a86c280f33" (UID: "dac7f002-d45b-415c-b77f-56a86c280f33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:11:59.848361 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.848261 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dac7f002-d45b-415c-b77f-56a86c280f33-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dac7f002-d45b-415c-b77f-56a86c280f33" (UID: "dac7f002-d45b-415c-b77f-56a86c280f33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:11:59.848430 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.848415 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac7f002-d45b-415c-b77f-56a86c280f33-kube-api-access-m5mt6" (OuterVolumeSpecName: "kube-api-access-m5mt6") pod "dac7f002-d45b-415c-b77f-56a86c280f33" (UID: "dac7f002-d45b-415c-b77f-56a86c280f33"). InnerVolumeSpecName "kube-api-access-m5mt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:11:59.947646 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.947612 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac7f002-d45b-415c-b77f-56a86c280f33-istio-token\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:11:59.947646 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.947641 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-istio-data\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:11:59.947646 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.947651 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac7f002-d45b-415c-b77f-56a86c280f33-istio-envoy\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:11:59.947897 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.947659 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac7f002-d45b-415c-b77f-56a86c280f33-istio-podinfo\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:11:59.947897 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.947670 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5mt6\" (UniqueName: \"kubernetes.io/projected/dac7f002-d45b-415c-b77f-56a86c280f33-kube-api-access-m5mt6\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:11:59.975395 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.975362 2585 generic.go:356] "Generic (PLEG): container finished" podID="dac7f002-d45b-415c-b77f-56a86c280f33" containerID="784d8590dabbbabad8aab35d2cf6571263b4795f89db79ac90855de8993b0134" exitCode=0 Jul 10 17:11:59.975546 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.975409 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" Jul 10 17:11:59.975546 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.975449 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" event={"ID":"dac7f002-d45b-415c-b77f-56a86c280f33","Type":"ContainerDied","Data":"784d8590dabbbabad8aab35d2cf6571263b4795f89db79ac90855de8993b0134"} Jul 10 17:11:59.975546 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.975511 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw" event={"ID":"dac7f002-d45b-415c-b77f-56a86c280f33","Type":"ContainerDied","Data":"d7aeca03d330ee84ec213a5875391d2e10251099ce07890a17b45594c4d7c3d2"} Jul 10 17:11:59.975546 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.975535 2585 scope.go:117] "RemoveContainer" containerID="784d8590dabbbabad8aab35d2cf6571263b4795f89db79ac90855de8993b0134" Jul 10 17:11:59.985620 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.985603 2585 scope.go:117] "RemoveContainer" containerID="6178123ebaf252b6cbff6efca461a2a02b221867096f09551567ccf1dc759587" Jul 10 17:11:59.993718 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.993700 2585 scope.go:117] "RemoveContainer" containerID="784d8590dabbbabad8aab35d2cf6571263b4795f89db79ac90855de8993b0134" Jul 10 17:11:59.994058 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:11:59.994036 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"784d8590dabbbabad8aab35d2cf6571263b4795f89db79ac90855de8993b0134\": container with ID starting with 784d8590dabbbabad8aab35d2cf6571263b4795f89db79ac90855de8993b0134 not found: ID does not exist" containerID="784d8590dabbbabad8aab35d2cf6571263b4795f89db79ac90855de8993b0134" Jul 10 17:11:59.994160 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.994070 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784d8590dabbbabad8aab35d2cf6571263b4795f89db79ac90855de8993b0134"} err="failed to get container status \"784d8590dabbbabad8aab35d2cf6571263b4795f89db79ac90855de8993b0134\": rpc error: code = NotFound desc = could not find container \"784d8590dabbbabad8aab35d2cf6571263b4795f89db79ac90855de8993b0134\": container with ID starting with 784d8590dabbbabad8aab35d2cf6571263b4795f89db79ac90855de8993b0134 not found: ID does not exist" Jul 10 17:11:59.994160 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.994095 2585 scope.go:117] "RemoveContainer" containerID="6178123ebaf252b6cbff6efca461a2a02b221867096f09551567ccf1dc759587" Jul 10 17:11:59.994378 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:11:59.994360 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6178123ebaf252b6cbff6efca461a2a02b221867096f09551567ccf1dc759587\": container with ID starting with 6178123ebaf252b6cbff6efca461a2a02b221867096f09551567ccf1dc759587 not found: ID does not exist" containerID="6178123ebaf252b6cbff6efca461a2a02b221867096f09551567ccf1dc759587" Jul 10 17:11:59.994432 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.994385 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6178123ebaf252b6cbff6efca461a2a02b221867096f09551567ccf1dc759587"} err="failed to get container status \"6178123ebaf252b6cbff6efca461a2a02b221867096f09551567ccf1dc759587\": rpc error: code = NotFound desc = could not find container \"6178123ebaf252b6cbff6efca461a2a02b221867096f09551567ccf1dc759587\": container with ID starting with 6178123ebaf252b6cbff6efca461a2a02b221867096f09551567ccf1dc759587 not found: ID does not exist" Jul 10 17:11:59.998550 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:11:59.998520 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw"] Jul 10 17:12:00.003454 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:00.003432 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e08efa8c-openshift-default-7b5c678bf9-l5llw"] Jul 10 17:12:01.326260 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:01.326229 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dac7f002-d45b-415c-b77f-56a86c280f33" path="/var/lib/kubelet/pods/dac7f002-d45b-415c-b77f-56a86c280f33/volumes" Jul 10 17:12:13.899307 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.899217 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj"] Jul 10 17:12:13.899921 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.899574 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac7f002-d45b-415c-b77f-56a86c280f33" containerName="istio-proxy" Jul 10 17:12:13.899921 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.899586 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac7f002-d45b-415c-b77f-56a86c280f33" containerName="istio-proxy" Jul 10 17:12:13.899921 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.899598 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac7f002-d45b-415c-b77f-56a86c280f33" containerName="istio-proxy" Jul 10 17:12:13.899921 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.899603 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac7f002-d45b-415c-b77f-56a86c280f33" containerName="istio-proxy" Jul 10 17:12:13.899921 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.899670 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea7c43bc-3db9-4209-96a7-54ed8a38b641" containerName="istio-proxy" Jul 10 17:12:13.899921 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.899681 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="dac7f002-d45b-415c-b77f-56a86c280f33" containerName="istio-proxy" Jul 10 17:12:13.902690 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.902671 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:13.906071 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.906048 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-af50a058-openshift-default-dockercfg-h9l52\"" Jul 10 17:12:13.926270 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.926241 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj"] Jul 10 17:12:13.988516 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.988478 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-data\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:13.988730 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.988523 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-credential-socket\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:13.988730 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.988591 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-token\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:13.988730 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.988616 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674572bb-8841-4e1e-80e6-6a4bd4bef924-istiod-ca-cert\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:13.988730 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.988655 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgzrq\" (UniqueName: \"kubernetes.io/projected/674572bb-8841-4e1e-80e6-6a4bd4bef924-kube-api-access-kgzrq\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:13.988984 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.988780 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-envoy\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:13.988984 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.988824 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-workload-certs\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:13.988984 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.988854 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-podinfo\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:13.988984 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:13.988900 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-workload-socket\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.089340 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.089300 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-workload-certs\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.089340 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.089344 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-podinfo\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.089600 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.089384 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-workload-socket\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.089600 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.089403 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-data\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.089600 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.089424 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-credential-socket\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.089600 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.089463 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-token\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.089844 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.089600 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674572bb-8841-4e1e-80e6-6a4bd4bef924-istiod-ca-cert\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.089844 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.089675 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgzrq\" (UniqueName: \"kubernetes.io/projected/674572bb-8841-4e1e-80e6-6a4bd4bef924-kube-api-access-kgzrq\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.089844 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.089736 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-workload-certs\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.089844 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.089778 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-envoy\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.089844 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.089811 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-workload-socket\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.090083 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.089920 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-data\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.090083 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.089953 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-credential-socket\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.090383 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.090365 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674572bb-8841-4e1e-80e6-6a4bd4bef924-istiod-ca-cert\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.091859 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.091841 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-envoy\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.091960 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.091945 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-podinfo\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.099556 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.099527 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgzrq\" (UniqueName: \"kubernetes.io/projected/674572bb-8841-4e1e-80e6-6a4bd4bef924-kube-api-access-kgzrq\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.099706 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.099555 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-token\") pod \"e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.213006 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.212909 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:14.360351 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.360323 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj"] Jul 10 17:12:14.362223 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:12:14.362192 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674572bb_8841_4e1e_80e6_6a4bd4bef924.slice/crio-fead5c88ad6f06b2f3ac2388be0a0fe22ef7f42559f783e698bab790590bb85c WatchSource:0}: Error finding container fead5c88ad6f06b2f3ac2388be0a0fe22ef7f42559f783e698bab790590bb85c: Status 404 returned error can't find the container with id fead5c88ad6f06b2f3ac2388be0a0fe22ef7f42559f783e698bab790590bb85c Jul 10 17:12:14.364420 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.364376 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:12:14.364509 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.364479 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:12:14.364549 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:14.364526 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:12:15.070090 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:15.070053 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" event={"ID":"674572bb-8841-4e1e-80e6-6a4bd4bef924","Type":"ContainerStarted","Data":"8219e732e906eb9de4f4b576df3ace14f015a804c1cb1fcfb6aa21c20c6cdf11"} Jul 10 17:12:15.070497 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:15.070143 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" event={"ID":"674572bb-8841-4e1e-80e6-6a4bd4bef924","Type":"ContainerStarted","Data":"fead5c88ad6f06b2f3ac2388be0a0fe22ef7f42559f783e698bab790590bb85c"} Jul 10 17:12:15.093369 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:15.093319 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" podStartSLOduration=2.093303659 podStartE2EDuration="2.093303659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:15.091702393 +0000 UTC m=+2154.513578888" watchObservedRunningTime="2026-07-10 17:12:15.093303659 +0000 UTC m=+2154.515180105" Jul 10 17:12:15.213804 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:15.213771 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:15.215140 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:15.215112 2585 patch_prober.go:28] interesting pod/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 10 17:12:15.215272 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:15.215175 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" podUID="674572bb-8841-4e1e-80e6-6a4bd4bef924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 10 17:12:16.217267 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:16.217234 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:17.083091 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:17.083047 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:17.084024 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:17.084005 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:22.904226 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:22.904184 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:12:28.197887 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.197843 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95"] Jul 10 17:12:28.198573 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.198553 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="dac7f002-d45b-415c-b77f-56a86c280f33" containerName="istio-proxy" Jul 10 17:12:28.201335 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.201312 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.205180 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.205153 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-af50a058\"/\"kube-root-ca.crt\"" Jul 10 17:12:28.206204 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.206183 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-af50a058\"/\"default-dockercfg-gpw99\"" Jul 10 17:12:28.206324 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.206189 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-af50a058\"/\"e2e-lifecycle-model-af50a058-kserve-self-signed-certs\"" Jul 10 17:12:28.206324 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.206223 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-af50a058\"/\"openshift-service-ca.crt\"" Jul 10 17:12:28.211875 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.211851 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95"] Jul 10 17:12:28.334342 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.334310 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-dshm\") pod \"e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.334342 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.334347 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-tmp-dir\") pod \"e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.334557 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.334454 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-model-cache\") pod \"e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.334557 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.334484 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l44w\" (UniqueName: \"kubernetes.io/projected/5587e8ee-89f9-4221-9c01-a2853c90ae6d-kube-api-access-5l44w\") pod \"e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.334557 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.334515 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5587e8ee-89f9-4221-9c01-a2853c90ae6d-tls-certs\") pod \"e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.334557 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.334550 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-home\") pod \"e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.435702 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.435601 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-model-cache\") pod \"e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.435882 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.435794 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l44w\" (UniqueName: \"kubernetes.io/projected/5587e8ee-89f9-4221-9c01-a2853c90ae6d-kube-api-access-5l44w\") pod \"e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.435882 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.435870 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5587e8ee-89f9-4221-9c01-a2853c90ae6d-tls-certs\") pod \"e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.435962 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.435928 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-home\") pod \"e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.436000 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.435980 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-dshm\") pod \"e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.436045 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.436030 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-tmp-dir\") pod \"e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.436107 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.436085 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-model-cache\") pod \"e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.436328 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.436300 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-home\") pod \"e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.436455 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.436360 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-tmp-dir\") pod \"e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.438233 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.438211 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-dshm\") pod \"e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.438400 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.438383 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5587e8ee-89f9-4221-9c01-a2853c90ae6d-tls-certs\") pod \"e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.445793 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.445769 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l44w\" (UniqueName: \"kubernetes.io/projected/5587e8ee-89f9-4221-9c01-a2853c90ae6d-kube-api-access-5l44w\") pod \"e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.514256 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.514214 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:28.666573 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:28.666541 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95"] Jul 10 17:12:28.668949 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:12:28.668914 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5587e8ee_89f9_4221_9c01_a2853c90ae6d.slice/crio-fc812e1399b5807c8d6654f451b456e05a926c58a6d98f12c912896bd2a1181b WatchSource:0}: Error finding container fc812e1399b5807c8d6654f451b456e05a926c58a6d98f12c912896bd2a1181b: Status 404 returned error can't find the container with id fc812e1399b5807c8d6654f451b456e05a926c58a6d98f12c912896bd2a1181b Jul 10 17:12:29.154074 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:29.154028 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" event={"ID":"5587e8ee-89f9-4221-9c01-a2853c90ae6d","Type":"ContainerStarted","Data":"f541f726ecbb0283094f3f352b37f827f965cdcdaa4179bf8ac8aae529127ace"} Jul 10 17:12:29.154273 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:29.154080 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" event={"ID":"5587e8ee-89f9-4221-9c01-a2853c90ae6d","Type":"ContainerStarted","Data":"fc812e1399b5807c8d6654f451b456e05a926c58a6d98f12c912896bd2a1181b"} Jul 10 17:12:29.174041 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:29.173977 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" podStartSLOduration=1.173955214 podStartE2EDuration="1.173955214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:29.172051675 +0000 UTC m=+2168.593928169" watchObservedRunningTime="2026-07-10 17:12:29.173955214 +0000 UTC m=+2168.595831715" Jul 10 17:12:31.167438 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:31.167402 2585 generic.go:356] "Generic (PLEG): container finished" podID="674572bb-8841-4e1e-80e6-6a4bd4bef924" containerID="8219e732e906eb9de4f4b576df3ace14f015a804c1cb1fcfb6aa21c20c6cdf11" exitCode=0 Jul 10 17:12:31.167885 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:31.167471 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" event={"ID":"674572bb-8841-4e1e-80e6-6a4bd4bef924","Type":"ContainerDied","Data":"8219e732e906eb9de4f4b576df3ace14f015a804c1cb1fcfb6aa21c20c6cdf11"} Jul 10 17:12:31.167936 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:31.167905 2585 scope.go:117] "RemoveContainer" containerID="8219e732e906eb9de4f4b576df3ace14f015a804c1cb1fcfb6aa21c20c6cdf11" Jul 10 17:12:31.168912 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:31.168885 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:12:31.168967 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:31.168954 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:12:31.169004 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:31.168986 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:12:32.083201 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:32.083140 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:32.174652 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:32.174614 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" event={"ID":"674572bb-8841-4e1e-80e6-6a4bd4bef924","Type":"ContainerStarted","Data":"d8d31ceb3bfc3c3206234c3b44ed0c110bddad818f3709c338322b730138f7a6"} Jul 10 17:12:32.213276 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:32.213239 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:32.219977 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:32.219941 2585 patch_prober.go:28] interesting pod/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 10 17:12:32.220164 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:32.220005 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" podUID="674572bb-8841-4e1e-80e6-6a4bd4bef924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 10 17:12:32.403712 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:32.403673 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:12:33.214149 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:33.214113 2585 patch_prober.go:28] interesting pod/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 10 17:12:33.214550 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:33.214172 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" podUID="674572bb-8841-4e1e-80e6-6a4bd4bef924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 10 17:12:34.213350 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:34.213310 2585 patch_prober.go:28] interesting pod/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 10 17:12:34.213533 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:34.213391 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" podUID="674572bb-8841-4e1e-80e6-6a4bd4bef924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 10 17:12:35.213683 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:35.213650 2585 patch_prober.go:28] interesting pod/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 10 17:12:35.214193 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:35.213727 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" podUID="674572bb-8841-4e1e-80e6-6a4bd4bef924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 10 17:12:36.217809 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:36.217777 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:36.218291 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:36.218179 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:36.219004 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:36.218982 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:12:38.514850 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:38.514802 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:38.515291 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:38.514863 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:38.527967 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:38.527928 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:39.224536 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:39.224501 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:12:46.408152 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:12:46.408089 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:13:03.794875 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:03.794836 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:13:09.507652 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:09.507623 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95"] Jul 10 17:13:21.259398 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:21.259365 2585 scope.go:117] "RemoveContainer" containerID="5b8d4937373050087201b40ec2eb7d3bf6e978630d3d9192363504888dc7306c" Jul 10 17:13:24.527818 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.527772 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" podUID="5587e8ee-89f9-4221-9c01-a2853c90ae6d" containerName="main" containerID="cri-o://f541f726ecbb0283094f3f352b37f827f965cdcdaa4179bf8ac8aae529127ace" gracePeriod=45 Jul 10 17:13:24.782283 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.782220 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:13:24.892384 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.892347 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5587e8ee-89f9-4221-9c01-a2853c90ae6d-tls-certs\") pod \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " Jul 10 17:13:24.892563 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.892404 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-home\") pod \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " Jul 10 17:13:24.892563 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.892450 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-tmp-dir\") pod \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " Jul 10 17:13:24.892563 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.892468 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-model-cache\") pod \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " Jul 10 17:13:24.892563 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.892510 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-dshm\") pod \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " Jul 10 17:13:24.892563 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.892536 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l44w\" (UniqueName: \"kubernetes.io/projected/5587e8ee-89f9-4221-9c01-a2853c90ae6d-kube-api-access-5l44w\") pod \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\" (UID: \"5587e8ee-89f9-4221-9c01-a2853c90ae6d\") " Jul 10 17:13:24.892865 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.892717 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "5587e8ee-89f9-4221-9c01-a2853c90ae6d" (UID: "5587e8ee-89f9-4221-9c01-a2853c90ae6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:13:24.892865 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.892779 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-home" (OuterVolumeSpecName: "home") pod "5587e8ee-89f9-4221-9c01-a2853c90ae6d" (UID: "5587e8ee-89f9-4221-9c01-a2853c90ae6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:13:24.892865 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.892795 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5587e8ee-89f9-4221-9c01-a2853c90ae6d" (UID: "5587e8ee-89f9-4221-9c01-a2853c90ae6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:13:24.892994 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.892930 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-home\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:13:24.892994 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.892947 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-tmp-dir\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:13:24.892994 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.892957 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-model-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:13:24.894638 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.894618 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5587e8ee-89f9-4221-9c01-a2853c90ae6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5587e8ee-89f9-4221-9c01-a2853c90ae6d" (UID: "5587e8ee-89f9-4221-9c01-a2853c90ae6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:13:24.894716 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.894700 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5587e8ee-89f9-4221-9c01-a2853c90ae6d-kube-api-access-5l44w" (OuterVolumeSpecName: "kube-api-access-5l44w") pod "5587e8ee-89f9-4221-9c01-a2853c90ae6d" (UID: "5587e8ee-89f9-4221-9c01-a2853c90ae6d"). InnerVolumeSpecName "kube-api-access-5l44w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:13:24.894783 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.894703 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-dshm" (OuterVolumeSpecName: "dshm") pod "5587e8ee-89f9-4221-9c01-a2853c90ae6d" (UID: "5587e8ee-89f9-4221-9c01-a2853c90ae6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:13:24.993796 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.993725 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5587e8ee-89f9-4221-9c01-a2853c90ae6d-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:13:24.993796 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.993793 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5587e8ee-89f9-4221-9c01-a2853c90ae6d-dshm\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:13:24.993796 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:24.993804 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l44w\" (UniqueName: \"kubernetes.io/projected/5587e8ee-89f9-4221-9c01-a2853c90ae6d-kube-api-access-5l44w\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:13:25.471226 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:25.471188 2585 generic.go:356] "Generic (PLEG): container finished" podID="5587e8ee-89f9-4221-9c01-a2853c90ae6d" containerID="f541f726ecbb0283094f3f352b37f827f965cdcdaa4179bf8ac8aae529127ace" exitCode=0 Jul 10 17:13:25.471226 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:25.471227 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" Jul 10 17:13:25.471429 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:25.471279 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" event={"ID":"5587e8ee-89f9-4221-9c01-a2853c90ae6d","Type":"ContainerDied","Data":"f541f726ecbb0283094f3f352b37f827f965cdcdaa4179bf8ac8aae529127ace"} Jul 10 17:13:25.471429 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:25.471314 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95" event={"ID":"5587e8ee-89f9-4221-9c01-a2853c90ae6d","Type":"ContainerDied","Data":"fc812e1399b5807c8d6654f451b456e05a926c58a6d98f12c912896bd2a1181b"} Jul 10 17:13:25.471429 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:25.471334 2585 scope.go:117] "RemoveContainer" containerID="f541f726ecbb0283094f3f352b37f827f965cdcdaa4179bf8ac8aae529127ace" Jul 10 17:13:25.480594 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:25.480573 2585 scope.go:117] "RemoveContainer" containerID="f541f726ecbb0283094f3f352b37f827f965cdcdaa4179bf8ac8aae529127ace" Jul 10 17:13:25.480856 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:13:25.480839 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f541f726ecbb0283094f3f352b37f827f965cdcdaa4179bf8ac8aae529127ace\": container with ID starting with f541f726ecbb0283094f3f352b37f827f965cdcdaa4179bf8ac8aae529127ace not found: ID does not exist" containerID="f541f726ecbb0283094f3f352b37f827f965cdcdaa4179bf8ac8aae529127ace" Jul 10 17:13:25.480908 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:25.480865 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f541f726ecbb0283094f3f352b37f827f965cdcdaa4179bf8ac8aae529127ace"} err="failed to get container status \"f541f726ecbb0283094f3f352b37f827f965cdcdaa4179bf8ac8aae529127ace\": rpc error: code = NotFound desc = could not find container \"f541f726ecbb0283094f3f352b37f827f965cdcdaa4179bf8ac8aae529127ace\": container with ID starting with f541f726ecbb0283094f3f352b37f827f965cdcdaa4179bf8ac8aae529127ace not found: ID does not exist" Jul 10 17:13:25.489383 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:25.489358 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95"] Jul 10 17:13:25.493490 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:25.493463 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-af50a058/e2e-lifecycle-model-af50a058-kserve-55bf478467-8jf95"] Jul 10 17:13:27.332647 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:13:27.332609 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5587e8ee-89f9-4221-9c01-a2853c90ae6d" path="/var/lib/kubelet/pods/5587e8ee-89f9-4221-9c01-a2853c90ae6d/volumes" Jul 10 17:14:16.726014 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:16.725961 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj"] Jul 10 17:14:16.726625 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:16.726245 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" podUID="674572bb-8841-4e1e-80e6-6a4bd4bef924" containerName="istio-proxy" containerID="cri-o://d8d31ceb3bfc3c3206234c3b44ed0c110bddad818f3709c338322b730138f7a6" gracePeriod=30 Jul 10 17:14:20.999469 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:20.999430 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:14:21.220150 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:21.220116 2585 patch_prober.go:28] interesting pod/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 17:14:21.220327 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:21.220204 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" podUID="674572bb-8841-4e1e-80e6-6a4bd4bef924" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 17:14:21.999108 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:21.999040 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:14:22.035071 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.035036 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-envoy\") pod \"674572bb-8841-4e1e-80e6-6a4bd4bef924\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " Jul 10 17:14:22.035551 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.035091 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674572bb-8841-4e1e-80e6-6a4bd4bef924-istiod-ca-cert\") pod \"674572bb-8841-4e1e-80e6-6a4bd4bef924\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " Jul 10 17:14:22.035551 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.035124 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-token\") pod \"674572bb-8841-4e1e-80e6-6a4bd4bef924\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " Jul 10 17:14:22.035551 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.035169 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-workload-certs\") pod \"674572bb-8841-4e1e-80e6-6a4bd4bef924\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " Jul 10 17:14:22.035551 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.035257 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-workload-socket\") pod \"674572bb-8841-4e1e-80e6-6a4bd4bef924\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " Jul 10 17:14:22.035551 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.035295 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-credential-socket\") pod \"674572bb-8841-4e1e-80e6-6a4bd4bef924\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " Jul 10 17:14:22.035551 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.035327 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-data\") pod \"674572bb-8841-4e1e-80e6-6a4bd4bef924\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " Jul 10 17:14:22.035551 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.035361 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgzrq\" (UniqueName: \"kubernetes.io/projected/674572bb-8841-4e1e-80e6-6a4bd4bef924-kube-api-access-kgzrq\") pod \"674572bb-8841-4e1e-80e6-6a4bd4bef924\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " Jul 10 17:14:22.035551 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.035410 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-podinfo\") pod \"674572bb-8841-4e1e-80e6-6a4bd4bef924\" (UID: \"674572bb-8841-4e1e-80e6-6a4bd4bef924\") " Jul 10 17:14:22.037124 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.036049 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "674572bb-8841-4e1e-80e6-6a4bd4bef924" (UID: "674572bb-8841-4e1e-80e6-6a4bd4bef924"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:14:22.037124 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.036386 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "674572bb-8841-4e1e-80e6-6a4bd4bef924" (UID: "674572bb-8841-4e1e-80e6-6a4bd4bef924"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:14:22.037124 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.037089 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-data" (OuterVolumeSpecName: "istio-data") pod "674572bb-8841-4e1e-80e6-6a4bd4bef924" (UID: "674572bb-8841-4e1e-80e6-6a4bd4bef924"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:14:22.037472 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.037437 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "674572bb-8841-4e1e-80e6-6a4bd4bef924" (UID: "674572bb-8841-4e1e-80e6-6a4bd4bef924"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:14:22.037762 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.037713 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/674572bb-8841-4e1e-80e6-6a4bd4bef924-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "674572bb-8841-4e1e-80e6-6a4bd4bef924" (UID: "674572bb-8841-4e1e-80e6-6a4bd4bef924"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:14:22.038306 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.038273 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-token" (OuterVolumeSpecName: "istio-token") pod "674572bb-8841-4e1e-80e6-6a4bd4bef924" (UID: "674572bb-8841-4e1e-80e6-6a4bd4bef924"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:14:22.038509 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.038476 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "674572bb-8841-4e1e-80e6-6a4bd4bef924" (UID: "674572bb-8841-4e1e-80e6-6a4bd4bef924"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:14:22.038698 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.038671 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "674572bb-8841-4e1e-80e6-6a4bd4bef924" (UID: "674572bb-8841-4e1e-80e6-6a4bd4bef924"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:14:22.040597 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.040570 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674572bb-8841-4e1e-80e6-6a4bd4bef924-kube-api-access-kgzrq" (OuterVolumeSpecName: "kube-api-access-kgzrq") pod "674572bb-8841-4e1e-80e6-6a4bd4bef924" (UID: "674572bb-8841-4e1e-80e6-6a4bd4bef924"). InnerVolumeSpecName "kube-api-access-kgzrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:14:22.136870 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.136831 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-workload-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:14:22.136870 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.136863 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-credential-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:14:22.136870 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.136873 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-data\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:14:22.137112 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.136882 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgzrq\" (UniqueName: \"kubernetes.io/projected/674572bb-8841-4e1e-80e6-6a4bd4bef924-kube-api-access-kgzrq\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:14:22.137112 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.136892 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-podinfo\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:14:22.137112 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.136900 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-envoy\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:14:22.137112 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.136908 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674572bb-8841-4e1e-80e6-6a4bd4bef924-istiod-ca-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:14:22.137112 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.136915 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/674572bb-8841-4e1e-80e6-6a4bd4bef924-istio-token\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:14:22.137112 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.136923 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674572bb-8841-4e1e-80e6-6a4bd4bef924-workload-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:14:22.781210 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.781173 2585 generic.go:356] "Generic (PLEG): container finished" podID="674572bb-8841-4e1e-80e6-6a4bd4bef924" containerID="d8d31ceb3bfc3c3206234c3b44ed0c110bddad818f3709c338322b730138f7a6" exitCode=0 Jul 10 17:14:22.781438 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.781242 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" event={"ID":"674572bb-8841-4e1e-80e6-6a4bd4bef924","Type":"ContainerDied","Data":"d8d31ceb3bfc3c3206234c3b44ed0c110bddad818f3709c338322b730138f7a6"} Jul 10 17:14:22.781438 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.781245 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" Jul 10 17:14:22.781438 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.781312 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj" event={"ID":"674572bb-8841-4e1e-80e6-6a4bd4bef924","Type":"ContainerDied","Data":"fead5c88ad6f06b2f3ac2388be0a0fe22ef7f42559f783e698bab790590bb85c"} Jul 10 17:14:22.781438 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.781354 2585 scope.go:117] "RemoveContainer" containerID="d8d31ceb3bfc3c3206234c3b44ed0c110bddad818f3709c338322b730138f7a6" Jul 10 17:14:22.794533 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.794507 2585 scope.go:117] "RemoveContainer" containerID="8219e732e906eb9de4f4b576df3ace14f015a804c1cb1fcfb6aa21c20c6cdf11" Jul 10 17:14:22.803255 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.803233 2585 scope.go:117] "RemoveContainer" containerID="d8d31ceb3bfc3c3206234c3b44ed0c110bddad818f3709c338322b730138f7a6" Jul 10 17:14:22.803526 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:14:22.803507 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d31ceb3bfc3c3206234c3b44ed0c110bddad818f3709c338322b730138f7a6\": container with ID starting with d8d31ceb3bfc3c3206234c3b44ed0c110bddad818f3709c338322b730138f7a6 not found: ID does not exist" containerID="d8d31ceb3bfc3c3206234c3b44ed0c110bddad818f3709c338322b730138f7a6" Jul 10 17:14:22.803587 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.803536 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d31ceb3bfc3c3206234c3b44ed0c110bddad818f3709c338322b730138f7a6"} err="failed to get container status \"d8d31ceb3bfc3c3206234c3b44ed0c110bddad818f3709c338322b730138f7a6\": rpc error: code = NotFound desc = could not find container \"d8d31ceb3bfc3c3206234c3b44ed0c110bddad818f3709c338322b730138f7a6\": container with ID starting with d8d31ceb3bfc3c3206234c3b44ed0c110bddad818f3709c338322b730138f7a6 not found: ID does not exist" Jul 10 17:14:22.803587 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.803556 2585 scope.go:117] "RemoveContainer" containerID="8219e732e906eb9de4f4b576df3ace14f015a804c1cb1fcfb6aa21c20c6cdf11" Jul 10 17:14:22.803812 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:14:22.803792 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8219e732e906eb9de4f4b576df3ace14f015a804c1cb1fcfb6aa21c20c6cdf11\": container with ID starting with 8219e732e906eb9de4f4b576df3ace14f015a804c1cb1fcfb6aa21c20c6cdf11 not found: ID does not exist" containerID="8219e732e906eb9de4f4b576df3ace14f015a804c1cb1fcfb6aa21c20c6cdf11" Jul 10 17:14:22.803864 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.803816 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8219e732e906eb9de4f4b576df3ace14f015a804c1cb1fcfb6aa21c20c6cdf11"} err="failed to get container status \"8219e732e906eb9de4f4b576df3ace14f015a804c1cb1fcfb6aa21c20c6cdf11\": rpc error: code = NotFound desc = could not find container \"8219e732e906eb9de4f4b576df3ace14f015a804c1cb1fcfb6aa21c20c6cdf11\": container with ID starting with 8219e732e906eb9de4f4b576df3ace14f015a804c1cb1fcfb6aa21c20c6cdf11 not found: ID does not exist" Jul 10 17:14:22.808435 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.808412 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj"] Jul 10 17:14:22.812285 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:22.812250 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-af50a058-openshift-default-5c56f7dbb7-2gzpj"] Jul 10 17:14:23.326046 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:23.326013 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="674572bb-8841-4e1e-80e6-6a4bd4bef924" path="/var/lib/kubelet/pods/674572bb-8841-4e1e-80e6-6a4bd4bef924/volumes" Jul 10 17:14:35.595790 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:14:35.595729 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:15:11.112450 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.112409 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68"] Jul 10 17:15:11.113096 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.113069 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="674572bb-8841-4e1e-80e6-6a4bd4bef924" containerName="istio-proxy" Jul 10 17:15:11.113096 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.113090 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="674572bb-8841-4e1e-80e6-6a4bd4bef924" containerName="istio-proxy" Jul 10 17:15:11.113224 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.113113 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="674572bb-8841-4e1e-80e6-6a4bd4bef924" containerName="istio-proxy" Jul 10 17:15:11.113224 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.113121 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="674572bb-8841-4e1e-80e6-6a4bd4bef924" containerName="istio-proxy" Jul 10 17:15:11.113224 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.113155 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5587e8ee-89f9-4221-9c01-a2853c90ae6d" containerName="main" Jul 10 17:15:11.113224 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.113164 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5587e8ee-89f9-4221-9c01-a2853c90ae6d" containerName="main" Jul 10 17:15:11.113427 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.113272 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5587e8ee-89f9-4221-9c01-a2853c90ae6d" containerName="main" Jul 10 17:15:11.113427 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.113286 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="674572bb-8841-4e1e-80e6-6a4bd4bef924" containerName="istio-proxy" Jul 10 17:15:11.113427 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.113301 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="674572bb-8841-4e1e-80e6-6a4bd4bef924" containerName="istio-proxy" Jul 10 17:15:11.115895 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.115870 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.118889 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.118864 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-889e72-openshift-default-dockercfg-g6lls\"" Jul 10 17:15:11.130492 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.130460 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68"] Jul 10 17:15:11.211074 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.211032 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-envoy\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.211074 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.211076 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-credential-socket\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.211335 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.211103 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-workload-certs\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.211335 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.211131 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-token\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.211335 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.211150 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-data\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.211335 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.211196 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n2r7\" (UniqueName: \"kubernetes.io/projected/a46bc621-2409-4309-8fa7-da6c1410c4d4-kube-api-access-6n2r7\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.211335 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.211236 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-workload-socket\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.211515 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.211327 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-podinfo\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.211515 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.211366 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a46bc621-2409-4309-8fa7-da6c1410c4d4-istiod-ca-cert\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.312645 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.312587 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n2r7\" (UniqueName: \"kubernetes.io/projected/a46bc621-2409-4309-8fa7-da6c1410c4d4-kube-api-access-6n2r7\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.312906 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.312661 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-workload-socket\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.312906 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.312707 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-podinfo\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.312906 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.312735 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a46bc621-2409-4309-8fa7-da6c1410c4d4-istiod-ca-cert\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.312906 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.312813 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-envoy\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.312906 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.312846 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-credential-socket\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.312906 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.312876 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-workload-certs\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.313226 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.312911 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-token\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.313226 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.312937 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-data\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.313322 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.313273 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-workload-socket\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.313379 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.313334 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-credential-socket\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.313448 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.313426 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-workload-certs\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.313500 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.313430 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-data\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.313696 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.313676 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a46bc621-2409-4309-8fa7-da6c1410c4d4-istiod-ca-cert\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.315416 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.315391 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-envoy\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.315671 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.315646 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-podinfo\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.322705 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.322640 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n2r7\" (UniqueName: \"kubernetes.io/projected/a46bc621-2409-4309-8fa7-da6c1410c4d4-kube-api-access-6n2r7\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.323102 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.323079 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-token\") pod \"e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.427529 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.427432 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:11.576974 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.576941 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68"] Jul 10 17:15:11.578836 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:15:11.578799 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46bc621_2409_4309_8fa7_da6c1410c4d4.slice/crio-c8e980c52f46243257f53d4240f4da52d1bc7e9b8a9cc811c1cf6957bc44ba3d WatchSource:0}: Error finding container c8e980c52f46243257f53d4240f4da52d1bc7e9b8a9cc811c1cf6957bc44ba3d: Status 404 returned error can't find the container with id c8e980c52f46243257f53d4240f4da52d1bc7e9b8a9cc811c1cf6957bc44ba3d Jul 10 17:15:11.581153 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.581109 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:15:11.581276 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.581207 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:15:11.581276 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:11.581258 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:15:12.048647 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:12.048605 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" event={"ID":"a46bc621-2409-4309-8fa7-da6c1410c4d4","Type":"ContainerStarted","Data":"152164ad370538108c1fe220c3df2a2ccbf7b95c7697cf46dbfd23cfadc24ea5"} Jul 10 17:15:12.048849 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:12.048668 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" event={"ID":"a46bc621-2409-4309-8fa7-da6c1410c4d4","Type":"ContainerStarted","Data":"c8e980c52f46243257f53d4240f4da52d1bc7e9b8a9cc811c1cf6957bc44ba3d"} Jul 10 17:15:12.072548 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:12.072494 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" podStartSLOduration=1.072475478 podStartE2EDuration="1.072475478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:15:12.070858983 +0000 UTC m=+2331.492735469" watchObservedRunningTime="2026-07-10 17:15:12.072475478 +0000 UTC m=+2331.494351944" Jul 10 17:15:12.427856 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:12.427819 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:12.432993 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:12.432967 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:13.054912 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:13.054877 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:13.055818 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:13.055796 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:19.016950 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.016912 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn"] Jul 10 17:15:19.019916 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.019888 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.024003 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.023966 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a0f0a2-openshift-default-dockercfg-qrf8t\"" Jul 10 17:15:19.036782 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.036707 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn"] Jul 10 17:15:19.095229 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.095182 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v26tc\" (UniqueName: \"kubernetes.io/projected/b979de36-45fa-419e-a288-bede67435380-kube-api-access-v26tc\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.095229 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.095233 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-workload-certs\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.095463 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.095304 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b979de36-45fa-419e-a288-bede67435380-istiod-ca-cert\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.095463 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.095367 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-istio-data\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.095463 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.095387 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-credential-socket\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.095463 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.095412 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-istio-envoy\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.095463 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.095443 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-workload-socket\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.095463 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.095457 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b979de36-45fa-419e-a288-bede67435380-istio-podinfo\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.095658 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.095489 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b979de36-45fa-419e-a288-bede67435380-istio-token\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.196580 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.196535 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-workload-socket\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.196580 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.196581 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b979de36-45fa-419e-a288-bede67435380-istio-podinfo\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.196909 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.196611 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b979de36-45fa-419e-a288-bede67435380-istio-token\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.196909 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.196686 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v26tc\" (UniqueName: \"kubernetes.io/projected/b979de36-45fa-419e-a288-bede67435380-kube-api-access-v26tc\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.196909 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.196720 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-workload-certs\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.196909 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.196794 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b979de36-45fa-419e-a288-bede67435380-istiod-ca-cert\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.196909 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.196852 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-istio-data\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.196909 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.196881 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-credential-socket\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.197224 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.196912 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-istio-envoy\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.197224 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.196995 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-workload-socket\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.197224 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.197124 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-workload-certs\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.197374 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.197222 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-istio-data\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.197374 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.197269 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-credential-socket\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.197539 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.197520 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b979de36-45fa-419e-a288-bede67435380-istiod-ca-cert\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.199127 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.199105 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-istio-envoy\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.199241 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.199148 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b979de36-45fa-419e-a288-bede67435380-istio-podinfo\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.205703 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.205677 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b979de36-45fa-419e-a288-bede67435380-istio-token\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.205877 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.205828 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v26tc\" (UniqueName: \"kubernetes.io/projected/b979de36-45fa-419e-a288-bede67435380-kube-api-access-v26tc\") pod \"e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.330959 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.330878 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:19.482472 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.482443 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn"] Jul 10 17:15:19.484367 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:15:19.484336 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb979de36_45fa_419e_a288_bede67435380.slice/crio-f8e4c50fae00d4e328edb6ab921c25ea02eb8c908e18733e489cfa6f29e423c4 WatchSource:0}: Error finding container f8e4c50fae00d4e328edb6ab921c25ea02eb8c908e18733e489cfa6f29e423c4: Status 404 returned error can't find the container with id f8e4c50fae00d4e328edb6ab921c25ea02eb8c908e18733e489cfa6f29e423c4 Jul 10 17:15:19.486795 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.486732 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:15:19.486880 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.486842 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:15:19.486918 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:19.486884 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:15:20.096287 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:20.096248 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" event={"ID":"b979de36-45fa-419e-a288-bede67435380","Type":"ContainerStarted","Data":"111d768bafb4d93c12b6e4cef36766c240738a8dc1676013c404cc142f5ecc3e"} Jul 10 17:15:20.096673 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:20.096319 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" event={"ID":"b979de36-45fa-419e-a288-bede67435380","Type":"ContainerStarted","Data":"f8e4c50fae00d4e328edb6ab921c25ea02eb8c908e18733e489cfa6f29e423c4"} Jul 10 17:15:20.119483 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:20.119421 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" podStartSLOduration=2.119399473 podStartE2EDuration="2.119399473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:15:20.119095733 +0000 UTC m=+2339.540972209" watchObservedRunningTime="2026-07-10 17:15:20.119399473 +0000 UTC m=+2339.541275941" Jul 10 17:15:20.331125 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:20.331086 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:20.336188 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:20.336142 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:21.101798 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:21.101769 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:21.102725 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:21.102701 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:21.596193 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:21.596143 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:15:29.096976 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:29.096925 2585 patch_prober.go:28] interesting pod/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 17:15:29.097420 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:29.096998 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" podUID="a46bc621-2409-4309-8fa7-da6c1410c4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 17:15:30.163031 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:30.162995 2585 generic.go:356] "Generic (PLEG): container finished" podID="a46bc621-2409-4309-8fa7-da6c1410c4d4" containerID="152164ad370538108c1fe220c3df2a2ccbf7b95c7697cf46dbfd23cfadc24ea5" exitCode=0 Jul 10 17:15:30.163462 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:30.163052 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" event={"ID":"a46bc621-2409-4309-8fa7-da6c1410c4d4","Type":"ContainerDied","Data":"152164ad370538108c1fe220c3df2a2ccbf7b95c7697cf46dbfd23cfadc24ea5"} Jul 10 17:15:30.163509 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:30.163467 2585 scope.go:117] "RemoveContainer" containerID="152164ad370538108c1fe220c3df2a2ccbf7b95c7697cf46dbfd23cfadc24ea5" Jul 10 17:15:30.164613 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:30.164585 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:15:30.164676 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:30.164651 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:15:30.164712 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:30.164684 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:15:31.174808 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:31.174767 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" event={"ID":"a46bc621-2409-4309-8fa7-da6c1410c4d4","Type":"ContainerStarted","Data":"dbcab91508b2ccee38e524f3e50f7a0b8082a35a0114706b7f2e28b2d5db0cd4"} Jul 10 17:15:31.428186 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:31.428077 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:31.428342 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:31.428208 2585 patch_prober.go:28] interesting pod/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 10 17:15:31.428342 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:31.428244 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" podUID="a46bc621-2409-4309-8fa7-da6c1410c4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 10 17:15:32.428314 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:32.428271 2585 patch_prober.go:28] interesting pod/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 10 17:15:32.428846 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:32.428333 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" podUID="a46bc621-2409-4309-8fa7-da6c1410c4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 10 17:15:33.428766 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:33.428715 2585 patch_prober.go:28] interesting pod/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 10 17:15:33.429238 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:33.428803 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" podUID="a46bc621-2409-4309-8fa7-da6c1410c4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 10 17:15:33.508532 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:33.508496 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:15:34.428840 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:34.428798 2585 patch_prober.go:28] interesting pod/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 10 17:15:34.429266 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:34.428877 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" podUID="a46bc621-2409-4309-8fa7-da6c1410c4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 10 17:15:35.431599 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.431564 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:35.432100 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.432080 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:35.433019 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.432999 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:15:35.697803 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.697717 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq"] Jul 10 17:15:35.701151 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.701128 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:35.704310 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.704276 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-889e72\"/\"openshift-service-ca.crt\"" Jul 10 17:15:35.704443 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.704396 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-889e72\"/\"test-model-889e72-kserve-self-signed-certs\"" Jul 10 17:15:35.704760 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.704729 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-889e72\"/\"default-dockercfg-89rvr\"" Jul 10 17:15:35.704760 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.704737 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-889e72\"/\"kube-root-ca.crt\"" Jul 10 17:15:35.711254 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.711234 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq"] Jul 10 17:15:35.787997 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.787961 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5bzh\" (UniqueName: \"kubernetes.io/projected/50a34080-615e-41a8-a4f5-6df0a5491c7a-kube-api-access-b5bzh\") pod \"test-model-889e72-kserve-64987c9967-5p2mq\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:35.788213 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.788007 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50a34080-615e-41a8-a4f5-6df0a5491c7a-tls-certs\") pod \"test-model-889e72-kserve-64987c9967-5p2mq\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:35.788213 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.788032 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-model-cache\") pod \"test-model-889e72-kserve-64987c9967-5p2mq\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:35.788213 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.788056 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-dshm\") pod \"test-model-889e72-kserve-64987c9967-5p2mq\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:35.788213 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.788191 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-tmp-dir\") pod \"test-model-889e72-kserve-64987c9967-5p2mq\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:35.788423 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.788273 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-home\") pod \"test-model-889e72-kserve-64987c9967-5p2mq\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:35.889332 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.889287 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5bzh\" (UniqueName: \"kubernetes.io/projected/50a34080-615e-41a8-a4f5-6df0a5491c7a-kube-api-access-b5bzh\") pod \"test-model-889e72-kserve-64987c9967-5p2mq\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:35.889539 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.889388 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50a34080-615e-41a8-a4f5-6df0a5491c7a-tls-certs\") pod \"test-model-889e72-kserve-64987c9967-5p2mq\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:35.889539 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.889441 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-model-cache\") pod \"test-model-889e72-kserve-64987c9967-5p2mq\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:35.889763 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.889717 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-dshm\") pod \"test-model-889e72-kserve-64987c9967-5p2mq\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:35.889898 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.889813 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-tmp-dir\") pod \"test-model-889e72-kserve-64987c9967-5p2mq\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:35.889898 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.889888 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-home\") pod \"test-model-889e72-kserve-64987c9967-5p2mq\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:35.890690 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.890522 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-model-cache\") pod \"test-model-889e72-kserve-64987c9967-5p2mq\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:35.890819 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.890786 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-home\") pod \"test-model-889e72-kserve-64987c9967-5p2mq\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:35.891450 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.891432 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-tmp-dir\") pod \"test-model-889e72-kserve-64987c9967-5p2mq\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:35.905496 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.905432 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50a34080-615e-41a8-a4f5-6df0a5491c7a-tls-certs\") pod \"test-model-889e72-kserve-64987c9967-5p2mq\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:35.905802 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.905722 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-dshm\") pod \"test-model-889e72-kserve-64987c9967-5p2mq\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:35.908153 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:35.908121 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5bzh\" (UniqueName: \"kubernetes.io/projected/50a34080-615e-41a8-a4f5-6df0a5491c7a-kube-api-access-b5bzh\") pod \"test-model-889e72-kserve-64987c9967-5p2mq\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:36.012023 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:36.011934 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:36.161552 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:36.161524 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq"] Jul 10 17:15:36.163343 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:15:36.163318 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50a34080_615e_41a8_a4f5_6df0a5491c7a.slice/crio-dd4ca2e53af877f6efbf71039d6cca587594513920fc169225d6570f1760fed5 WatchSource:0}: Error finding container dd4ca2e53af877f6efbf71039d6cca587594513920fc169225d6570f1760fed5: Status 404 returned error can't find the container with id dd4ca2e53af877f6efbf71039d6cca587594513920fc169225d6570f1760fed5 Jul 10 17:15:36.207645 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:36.207616 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" event={"ID":"50a34080-615e-41a8-a4f5-6df0a5491c7a","Type":"ContainerStarted","Data":"dd4ca2e53af877f6efbf71039d6cca587594513920fc169225d6570f1760fed5"} Jul 10 17:15:36.935472 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:36.935023 2585 patch_prober.go:28] interesting pod/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:53108->10.133.0.55:15021: read: connection reset by peer" start-of-body= Jul 10 17:15:36.935472 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:36.935086 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" podUID="b979de36-45fa-419e-a288-bede67435380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:53108->10.133.0.55:15021: read: connection reset by peer" Jul 10 17:15:37.214697 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:37.214590 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" event={"ID":"50a34080-615e-41a8-a4f5-6df0a5491c7a","Type":"ContainerStarted","Data":"8f46497c8d89c1781f17c4022ec168277b9df770847412843327c1ca5bb69aca"} Jul 10 17:15:37.216335 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:37.216308 2585 generic.go:356] "Generic (PLEG): container finished" podID="b979de36-45fa-419e-a288-bede67435380" containerID="111d768bafb4d93c12b6e4cef36766c240738a8dc1676013c404cc142f5ecc3e" exitCode=0 Jul 10 17:15:37.216486 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:37.216387 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" event={"ID":"b979de36-45fa-419e-a288-bede67435380","Type":"ContainerDied","Data":"111d768bafb4d93c12b6e4cef36766c240738a8dc1676013c404cc142f5ecc3e"} Jul 10 17:15:37.216835 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:37.216818 2585 scope.go:117] "RemoveContainer" containerID="111d768bafb4d93c12b6e4cef36766c240738a8dc1676013c404cc142f5ecc3e" Jul 10 17:15:37.217797 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:37.217769 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:15:37.217904 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:37.217851 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:15:37.217904 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:37.217890 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:15:37.234494 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:37.234455 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" podStartSLOduration=2.234440705 podStartE2EDuration="2.234440705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:15:37.233245944 +0000 UTC m=+2356.655122434" watchObservedRunningTime="2026-07-10 17:15:37.234440705 +0000 UTC m=+2356.656317174" Jul 10 17:15:38.224502 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:38.224465 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" event={"ID":"b979de36-45fa-419e-a288-bede67435380","Type":"ContainerStarted","Data":"d75b0437ff1a870e3e28376a73f862937234e4a4fc0e33eab769e3de8f7df215"} Jul 10 17:15:38.331757 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:38.331724 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:38.331988 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:38.331966 2585 patch_prober.go:28] interesting pod/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 10 17:15:38.332032 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:38.332016 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" podUID="b979de36-45fa-419e-a288-bede67435380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 10 17:15:39.331516 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:39.331487 2585 patch_prober.go:28] interesting pod/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 10 17:15:39.331939 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:39.331535 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" podUID="b979de36-45fa-419e-a288-bede67435380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 10 17:15:40.331300 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:40.331257 2585 patch_prober.go:28] interesting pod/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 10 17:15:40.331468 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:40.331318 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" podUID="b979de36-45fa-419e-a288-bede67435380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 10 17:15:41.331706 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:41.331670 2585 patch_prober.go:28] interesting pod/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 10 17:15:41.332179 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:41.331728 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" podUID="b979de36-45fa-419e-a288-bede67435380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 10 17:15:41.704029 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:41.703988 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:15:42.335097 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:42.335059 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:42.335581 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:42.335438 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:42.336243 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:42.336222 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:15:46.012864 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:46.012831 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:46.012864 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:46.012875 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:46.025542 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:46.025513 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:15:46.283882 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:15:46.283796 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:16:02.799516 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:02.799483 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:16:19.777871 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:19.777835 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq"] Jul 10 17:16:21.219028 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:21.218998 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 17:16:21.230581 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:21.230554 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 17:16:23.491949 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:23.491898 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:16:34.799015 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:34.798969 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" podUID="50a34080-615e-41a8-a4f5-6df0a5491c7a" containerName="main" containerID="cri-o://8f46497c8d89c1781f17c4022ec168277b9df770847412843327c1ca5bb69aca" gracePeriod=45 Jul 10 17:16:35.061369 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.061291 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:16:35.097873 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.097839 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-dshm\") pod \"50a34080-615e-41a8-a4f5-6df0a5491c7a\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " Jul 10 17:16:35.098066 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.097917 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50a34080-615e-41a8-a4f5-6df0a5491c7a-tls-certs\") pod \"50a34080-615e-41a8-a4f5-6df0a5491c7a\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " Jul 10 17:16:35.098066 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.097945 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-model-cache\") pod \"50a34080-615e-41a8-a4f5-6df0a5491c7a\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " Jul 10 17:16:35.098298 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.098275 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-model-cache" (OuterVolumeSpecName: "model-cache") pod "50a34080-615e-41a8-a4f5-6df0a5491c7a" (UID: "50a34080-615e-41a8-a4f5-6df0a5491c7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:16:35.100078 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.100044 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-dshm" (OuterVolumeSpecName: "dshm") pod "50a34080-615e-41a8-a4f5-6df0a5491c7a" (UID: "50a34080-615e-41a8-a4f5-6df0a5491c7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:16:35.100288 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.100259 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50a34080-615e-41a8-a4f5-6df0a5491c7a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50a34080-615e-41a8-a4f5-6df0a5491c7a" (UID: "50a34080-615e-41a8-a4f5-6df0a5491c7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:16:35.198628 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.198586 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-home\") pod \"50a34080-615e-41a8-a4f5-6df0a5491c7a\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " Jul 10 17:16:35.198628 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.198631 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-tmp-dir\") pod \"50a34080-615e-41a8-a4f5-6df0a5491c7a\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " Jul 10 17:16:35.198930 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.198655 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5bzh\" (UniqueName: \"kubernetes.io/projected/50a34080-615e-41a8-a4f5-6df0a5491c7a-kube-api-access-b5bzh\") pod \"50a34080-615e-41a8-a4f5-6df0a5491c7a\" (UID: \"50a34080-615e-41a8-a4f5-6df0a5491c7a\") " Jul 10 17:16:35.198930 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.198870 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50a34080-615e-41a8-a4f5-6df0a5491c7a-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:16:35.198930 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.198886 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-model-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:16:35.198930 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.198895 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-dshm\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:16:35.198930 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.198899 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50a34080-615e-41a8-a4f5-6df0a5491c7a" (UID: "50a34080-615e-41a8-a4f5-6df0a5491c7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:16:35.198930 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.198906 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-home" (OuterVolumeSpecName: "home") pod "50a34080-615e-41a8-a4f5-6df0a5491c7a" (UID: "50a34080-615e-41a8-a4f5-6df0a5491c7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:16:35.200886 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.200855 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a34080-615e-41a8-a4f5-6df0a5491c7a-kube-api-access-b5bzh" (OuterVolumeSpecName: "kube-api-access-b5bzh") pod "50a34080-615e-41a8-a4f5-6df0a5491c7a" (UID: "50a34080-615e-41a8-a4f5-6df0a5491c7a"). InnerVolumeSpecName "kube-api-access-b5bzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:16:35.299532 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.299495 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-home\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:16:35.299532 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.299528 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50a34080-615e-41a8-a4f5-6df0a5491c7a-tmp-dir\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:16:35.299739 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.299540 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5bzh\" (UniqueName: \"kubernetes.io/projected/50a34080-615e-41a8-a4f5-6df0a5491c7a-kube-api-access-b5bzh\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:16:35.559899 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.559864 2585 generic.go:356] "Generic (PLEG): container finished" podID="50a34080-615e-41a8-a4f5-6df0a5491c7a" containerID="8f46497c8d89c1781f17c4022ec168277b9df770847412843327c1ca5bb69aca" exitCode=0 Jul 10 17:16:35.560096 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.559912 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" Jul 10 17:16:35.560096 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.559915 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" event={"ID":"50a34080-615e-41a8-a4f5-6df0a5491c7a","Type":"ContainerDied","Data":"8f46497c8d89c1781f17c4022ec168277b9df770847412843327c1ca5bb69aca"} Jul 10 17:16:35.560096 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.560021 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq" event={"ID":"50a34080-615e-41a8-a4f5-6df0a5491c7a","Type":"ContainerDied","Data":"dd4ca2e53af877f6efbf71039d6cca587594513920fc169225d6570f1760fed5"} Jul 10 17:16:35.560096 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.560037 2585 scope.go:117] "RemoveContainer" containerID="8f46497c8d89c1781f17c4022ec168277b9df770847412843327c1ca5bb69aca" Jul 10 17:16:35.569349 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.569314 2585 scope.go:117] "RemoveContainer" containerID="8f46497c8d89c1781f17c4022ec168277b9df770847412843327c1ca5bb69aca" Jul 10 17:16:35.569663 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:16:35.569641 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f46497c8d89c1781f17c4022ec168277b9df770847412843327c1ca5bb69aca\": container with ID starting with 8f46497c8d89c1781f17c4022ec168277b9df770847412843327c1ca5bb69aca not found: ID does not exist" containerID="8f46497c8d89c1781f17c4022ec168277b9df770847412843327c1ca5bb69aca" Jul 10 17:16:35.569711 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.569675 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f46497c8d89c1781f17c4022ec168277b9df770847412843327c1ca5bb69aca"} err="failed to get container status \"8f46497c8d89c1781f17c4022ec168277b9df770847412843327c1ca5bb69aca\": rpc error: code = NotFound desc = could not find container \"8f46497c8d89c1781f17c4022ec168277b9df770847412843327c1ca5bb69aca\": container with ID starting with 8f46497c8d89c1781f17c4022ec168277b9df770847412843327c1ca5bb69aca not found: ID does not exist" Jul 10 17:16:35.579190 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.579160 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq"] Jul 10 17:16:35.583463 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:35.583393 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-889e72/test-model-889e72-kserve-64987c9967-5p2mq"] Jul 10 17:16:37.326923 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:37.326884 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50a34080-615e-41a8-a4f5-6df0a5491c7a" path="/var/lib/kubelet/pods/50a34080-615e-41a8-a4f5-6df0a5491c7a/volumes" Jul 10 17:16:41.106105 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:41.106069 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:16:46.584266 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:46.584181 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68"] Jul 10 17:16:46.584659 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:46.584459 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" podUID="a46bc621-2409-4309-8fa7-da6c1410c4d4" containerName="istio-proxy" containerID="cri-o://dbcab91508b2ccee38e524f3e50f7a0b8082a35a0114706b7f2e28b2d5db0cd4" gracePeriod=30 Jul 10 17:16:50.434415 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:50.434378 2585 patch_prober.go:28] interesting pod/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 17:16:50.434825 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:50.434438 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" podUID="a46bc621-2409-4309-8fa7-da6c1410c4d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 17:16:51.835129 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.835095 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:16:51.953295 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.953271 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-data\") pod \"a46bc621-2409-4309-8fa7-da6c1410c4d4\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " Jul 10 17:16:51.953459 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.953354 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-podinfo\") pod \"a46bc621-2409-4309-8fa7-da6c1410c4d4\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " Jul 10 17:16:51.953459 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.953376 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n2r7\" (UniqueName: \"kubernetes.io/projected/a46bc621-2409-4309-8fa7-da6c1410c4d4-kube-api-access-6n2r7\") pod \"a46bc621-2409-4309-8fa7-da6c1410c4d4\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " Jul 10 17:16:51.953459 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.953398 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a46bc621-2409-4309-8fa7-da6c1410c4d4-istiod-ca-cert\") pod \"a46bc621-2409-4309-8fa7-da6c1410c4d4\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " Jul 10 17:16:51.953459 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.953421 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-workload-socket\") pod \"a46bc621-2409-4309-8fa7-da6c1410c4d4\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " Jul 10 17:16:51.953683 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.953507 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-token\") pod \"a46bc621-2409-4309-8fa7-da6c1410c4d4\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " Jul 10 17:16:51.953683 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.953556 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-credential-socket\") pod \"a46bc621-2409-4309-8fa7-da6c1410c4d4\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " Jul 10 17:16:51.953683 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.953603 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-envoy\") pod \"a46bc621-2409-4309-8fa7-da6c1410c4d4\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " Jul 10 17:16:51.953683 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.953630 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-workload-certs\") pod \"a46bc621-2409-4309-8fa7-da6c1410c4d4\" (UID: \"a46bc621-2409-4309-8fa7-da6c1410c4d4\") " Jul 10 17:16:51.953921 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.953692 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a46bc621-2409-4309-8fa7-da6c1410c4d4" (UID: "a46bc621-2409-4309-8fa7-da6c1410c4d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:16:51.953921 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.953824 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a46bc621-2409-4309-8fa7-da6c1410c4d4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a46bc621-2409-4309-8fa7-da6c1410c4d4" (UID: "a46bc621-2409-4309-8fa7-da6c1410c4d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:16:51.954029 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.953931 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a46bc621-2409-4309-8fa7-da6c1410c4d4" (UID: "a46bc621-2409-4309-8fa7-da6c1410c4d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:16:51.954097 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.954027 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a46bc621-2409-4309-8fa7-da6c1410c4d4" (UID: "a46bc621-2409-4309-8fa7-da6c1410c4d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:16:51.954160 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.954103 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-data" (OuterVolumeSpecName: "istio-data") pod "a46bc621-2409-4309-8fa7-da6c1410c4d4" (UID: "a46bc621-2409-4309-8fa7-da6c1410c4d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:16:51.954217 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.954175 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-workload-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:16:51.954217 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.954192 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-data\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:16:51.954217 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.954200 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a46bc621-2409-4309-8fa7-da6c1410c4d4-istiod-ca-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:16:51.954217 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.954208 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-workload-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:16:51.954416 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.954221 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-credential-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:16:51.955641 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.955610 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a46bc621-2409-4309-8fa7-da6c1410c4d4" (UID: "a46bc621-2409-4309-8fa7-da6c1410c4d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:16:51.955810 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.955790 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a46bc621-2409-4309-8fa7-da6c1410c4d4" (UID: "a46bc621-2409-4309-8fa7-da6c1410c4d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:16:51.956008 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.955980 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a46bc621-2409-4309-8fa7-da6c1410c4d4-kube-api-access-6n2r7" (OuterVolumeSpecName: "kube-api-access-6n2r7") pod "a46bc621-2409-4309-8fa7-da6c1410c4d4" (UID: "a46bc621-2409-4309-8fa7-da6c1410c4d4"). InnerVolumeSpecName "kube-api-access-6n2r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:16:51.956008 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:51.955988 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-token" (OuterVolumeSpecName: "istio-token") pod "a46bc621-2409-4309-8fa7-da6c1410c4d4" (UID: "a46bc621-2409-4309-8fa7-da6c1410c4d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:16:52.054731 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:52.054709 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-podinfo\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:16:52.054731 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:52.054730 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n2r7\" (UniqueName: \"kubernetes.io/projected/a46bc621-2409-4309-8fa7-da6c1410c4d4-kube-api-access-6n2r7\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:16:52.054883 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:52.054764 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-token\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:16:52.054883 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:52.054774 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a46bc621-2409-4309-8fa7-da6c1410c4d4-istio-envoy\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:16:52.660088 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:52.660047 2585 generic.go:356] "Generic (PLEG): container finished" podID="a46bc621-2409-4309-8fa7-da6c1410c4d4" containerID="dbcab91508b2ccee38e524f3e50f7a0b8082a35a0114706b7f2e28b2d5db0cd4" exitCode=0 Jul 10 17:16:52.660301 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:52.660089 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" event={"ID":"a46bc621-2409-4309-8fa7-da6c1410c4d4","Type":"ContainerDied","Data":"dbcab91508b2ccee38e524f3e50f7a0b8082a35a0114706b7f2e28b2d5db0cd4"} Jul 10 17:16:52.660301 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:52.660130 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" Jul 10 17:16:52.660301 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:52.660171 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68" event={"ID":"a46bc621-2409-4309-8fa7-da6c1410c4d4","Type":"ContainerDied","Data":"c8e980c52f46243257f53d4240f4da52d1bc7e9b8a9cc811c1cf6957bc44ba3d"} Jul 10 17:16:52.660301 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:52.660204 2585 scope.go:117] "RemoveContainer" containerID="dbcab91508b2ccee38e524f3e50f7a0b8082a35a0114706b7f2e28b2d5db0cd4" Jul 10 17:16:52.673439 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:52.673407 2585 scope.go:117] "RemoveContainer" containerID="152164ad370538108c1fe220c3df2a2ccbf7b95c7697cf46dbfd23cfadc24ea5" Jul 10 17:16:52.681939 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:52.681922 2585 scope.go:117] "RemoveContainer" containerID="dbcab91508b2ccee38e524f3e50f7a0b8082a35a0114706b7f2e28b2d5db0cd4" Jul 10 17:16:52.682168 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:16:52.682149 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbcab91508b2ccee38e524f3e50f7a0b8082a35a0114706b7f2e28b2d5db0cd4\": container with ID starting with dbcab91508b2ccee38e524f3e50f7a0b8082a35a0114706b7f2e28b2d5db0cd4 not found: ID does not exist" containerID="dbcab91508b2ccee38e524f3e50f7a0b8082a35a0114706b7f2e28b2d5db0cd4" Jul 10 17:16:52.682213 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:52.682176 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbcab91508b2ccee38e524f3e50f7a0b8082a35a0114706b7f2e28b2d5db0cd4"} err="failed to get container status \"dbcab91508b2ccee38e524f3e50f7a0b8082a35a0114706b7f2e28b2d5db0cd4\": rpc error: code = NotFound desc = could not find container \"dbcab91508b2ccee38e524f3e50f7a0b8082a35a0114706b7f2e28b2d5db0cd4\": container with ID starting with dbcab91508b2ccee38e524f3e50f7a0b8082a35a0114706b7f2e28b2d5db0cd4 not found: ID does not exist" Jul 10 17:16:52.682213 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:52.682193 2585 scope.go:117] "RemoveContainer" containerID="152164ad370538108c1fe220c3df2a2ccbf7b95c7697cf46dbfd23cfadc24ea5" Jul 10 17:16:52.682423 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:16:52.682409 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"152164ad370538108c1fe220c3df2a2ccbf7b95c7697cf46dbfd23cfadc24ea5\": container with ID starting with 152164ad370538108c1fe220c3df2a2ccbf7b95c7697cf46dbfd23cfadc24ea5 not found: ID does not exist" containerID="152164ad370538108c1fe220c3df2a2ccbf7b95c7697cf46dbfd23cfadc24ea5" Jul 10 17:16:52.682464 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:52.682427 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152164ad370538108c1fe220c3df2a2ccbf7b95c7697cf46dbfd23cfadc24ea5"} err="failed to get container status \"152164ad370538108c1fe220c3df2a2ccbf7b95c7697cf46dbfd23cfadc24ea5\": rpc error: code = NotFound desc = could not find container \"152164ad370538108c1fe220c3df2a2ccbf7b95c7697cf46dbfd23cfadc24ea5\": container with ID starting with 152164ad370538108c1fe220c3df2a2ccbf7b95c7697cf46dbfd23cfadc24ea5 not found: ID does not exist" Jul 10 17:16:52.691059 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:52.691038 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68"] Jul 10 17:16:52.696027 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:52.696005 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-889e72-openshift-default-6889bfbbcf-d5v68"] Jul 10 17:16:53.325852 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:16:53.325822 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a46bc621-2409-4309-8fa7-da6c1410c4d4" path="/var/lib/kubelet/pods/a46bc621-2409-4309-8fa7-da6c1410c4d4/volumes" Jul 10 17:17:00.397208 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:00.397172 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:17:12.699020 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:12.698981 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn"] Jul 10 17:17:12.699575 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:12.699221 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" podUID="b979de36-45fa-419e-a288-bede67435380" containerName="istio-proxy" containerID="cri-o://d75b0437ff1a870e3e28376a73f862937234e4a4fc0e33eab769e3de8f7df215" gracePeriod=30 Jul 10 17:17:13.582262 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.582217 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb"] Jul 10 17:17:13.582710 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.582689 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50a34080-615e-41a8-a4f5-6df0a5491c7a" containerName="main" Jul 10 17:17:13.582840 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.582713 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="50a34080-615e-41a8-a4f5-6df0a5491c7a" containerName="main" Jul 10 17:17:13.582840 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.582769 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a46bc621-2409-4309-8fa7-da6c1410c4d4" containerName="istio-proxy" Jul 10 17:17:13.582840 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.582780 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a46bc621-2409-4309-8fa7-da6c1410c4d4" containerName="istio-proxy" Jul 10 17:17:13.582840 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.582803 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a46bc621-2409-4309-8fa7-da6c1410c4d4" containerName="istio-proxy" Jul 10 17:17:13.582840 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.582812 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a46bc621-2409-4309-8fa7-da6c1410c4d4" containerName="istio-proxy" Jul 10 17:17:13.583074 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.582924 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a46bc621-2409-4309-8fa7-da6c1410c4d4" containerName="istio-proxy" Jul 10 17:17:13.583074 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.582941 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a46bc621-2409-4309-8fa7-da6c1410c4d4" containerName="istio-proxy" Jul 10 17:17:13.583074 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.582950 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="50a34080-615e-41a8-a4f5-6df0a5491c7a" containerName="main" Jul 10 17:17:13.585581 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.585552 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.588425 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.588396 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5e0f6f-openshift-default-dockercfg-6llgz\"" Jul 10 17:17:13.600659 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.600622 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb"] Jul 10 17:17:13.764070 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.764018 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-token\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.764070 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.764072 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76420a4e-6c81-4db7-bced-57e7237fd7b2-istiod-ca-cert\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.764614 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.764191 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-workload-socket\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.764614 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.764256 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-envoy\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.764614 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.764288 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-workload-certs\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.764614 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.764306 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttgvc\" (UniqueName: \"kubernetes.io/projected/76420a4e-6c81-4db7-bced-57e7237fd7b2-kube-api-access-ttgvc\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.764614 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.764367 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-credential-socket\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.764614 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.764395 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-data\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.764614 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.764451 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-podinfo\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.865105 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.865066 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-podinfo\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.865325 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.865157 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-token\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.865325 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.865175 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76420a4e-6c81-4db7-bced-57e7237fd7b2-istiod-ca-cert\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.865325 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.865206 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-workload-socket\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.865325 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.865248 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-envoy\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.865325 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.865278 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-workload-certs\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.865325 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.865312 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttgvc\" (UniqueName: \"kubernetes.io/projected/76420a4e-6c81-4db7-bced-57e7237fd7b2-kube-api-access-ttgvc\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.865625 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.865345 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-credential-socket\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.865625 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.865375 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-data\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.865799 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.865766 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-workload-certs\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.865945 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.865840 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-data\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.865945 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.865925 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-credential-socket\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.866061 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.866037 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-workload-socket\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.866169 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.866153 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76420a4e-6c81-4db7-bced-57e7237fd7b2-istiod-ca-cert\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.867640 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.867618 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-envoy\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.867719 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.867673 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-podinfo\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.878929 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.878894 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttgvc\" (UniqueName: \"kubernetes.io/projected/76420a4e-6c81-4db7-bced-57e7237fd7b2-kube-api-access-ttgvc\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.880726 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.880705 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-token\") pod \"e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:13.897795 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:13.897736 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:14.041650 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:14.041623 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb"] Jul 10 17:17:14.043179 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:17:14.043151 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76420a4e_6c81_4db7_bced_57e7237fd7b2.slice/crio-d78b7ab05e05e6f90138f43625ec5ec7ef5e2f1d3dba549eceb0a05940bdd80c WatchSource:0}: Error finding container d78b7ab05e05e6f90138f43625ec5ec7ef5e2f1d3dba549eceb0a05940bdd80c: Status 404 returned error can't find the container with id d78b7ab05e05e6f90138f43625ec5ec7ef5e2f1d3dba549eceb0a05940bdd80c Jul 10 17:17:14.045336 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:14.045299 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:17:14.045439 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:14.045388 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:17:14.045439 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:14.045431 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:17:14.787771 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:14.787707 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" event={"ID":"76420a4e-6c81-4db7-bced-57e7237fd7b2","Type":"ContainerStarted","Data":"5cc062d6293842020f1b0fe6c4cf95eddf0f257fdab84c30e5ade66582c65bda"} Jul 10 17:17:14.788245 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:14.787790 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" event={"ID":"76420a4e-6c81-4db7-bced-57e7237fd7b2","Type":"ContainerStarted","Data":"d78b7ab05e05e6f90138f43625ec5ec7ef5e2f1d3dba549eceb0a05940bdd80c"} Jul 10 17:17:14.814257 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:14.814200 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" podStartSLOduration=1.814182406 podStartE2EDuration="1.814182406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:17:14.813052692 +0000 UTC m=+2454.234929163" watchObservedRunningTime="2026-07-10 17:17:14.814182406 +0000 UTC m=+2454.236058870" Jul 10 17:17:14.898719 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:14.898679 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:15.904344 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:15.904306 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:16.799707 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:16.799673 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:16.800847 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:16.800818 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:17.956299 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:17.956274 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:17:18.003458 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.003355 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v26tc\" (UniqueName: \"kubernetes.io/projected/b979de36-45fa-419e-a288-bede67435380-kube-api-access-v26tc\") pod \"b979de36-45fa-419e-a288-bede67435380\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " Jul 10 17:17:18.003458 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.003408 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b979de36-45fa-419e-a288-bede67435380-istiod-ca-cert\") pod \"b979de36-45fa-419e-a288-bede67435380\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " Jul 10 17:17:18.003458 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.003447 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-workload-socket\") pod \"b979de36-45fa-419e-a288-bede67435380\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " Jul 10 17:17:18.003775 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.003477 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b979de36-45fa-419e-a288-bede67435380-istio-token\") pod \"b979de36-45fa-419e-a288-bede67435380\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " Jul 10 17:17:18.003775 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.003505 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-workload-certs\") pod \"b979de36-45fa-419e-a288-bede67435380\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " Jul 10 17:17:18.003775 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.003530 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b979de36-45fa-419e-a288-bede67435380-istio-podinfo\") pod \"b979de36-45fa-419e-a288-bede67435380\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " Jul 10 17:17:18.003775 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.003567 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-istio-data\") pod \"b979de36-45fa-419e-a288-bede67435380\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " Jul 10 17:17:18.003775 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.003623 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-istio-envoy\") pod \"b979de36-45fa-419e-a288-bede67435380\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " Jul 10 17:17:18.003775 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.003696 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-credential-socket\") pod \"b979de36-45fa-419e-a288-bede67435380\" (UID: \"b979de36-45fa-419e-a288-bede67435380\") " Jul 10 17:17:18.004078 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.003831 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b979de36-45fa-419e-a288-bede67435380" (UID: "b979de36-45fa-419e-a288-bede67435380"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:17:18.004078 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.003845 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b979de36-45fa-419e-a288-bede67435380" (UID: "b979de36-45fa-419e-a288-bede67435380"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:17:18.004219 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.004087 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-workload-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:17:18.004219 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.004125 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-workload-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:17:18.004219 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.004082 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b979de36-45fa-419e-a288-bede67435380" (UID: "b979de36-45fa-419e-a288-bede67435380"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:17:18.004387 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.004257 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b979de36-45fa-419e-a288-bede67435380-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b979de36-45fa-419e-a288-bede67435380" (UID: "b979de36-45fa-419e-a288-bede67435380"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:17:18.004655 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.004626 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-istio-data" (OuterVolumeSpecName: "istio-data") pod "b979de36-45fa-419e-a288-bede67435380" (UID: "b979de36-45fa-419e-a288-bede67435380"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:17:18.006191 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.006158 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b979de36-45fa-419e-a288-bede67435380" (UID: "b979de36-45fa-419e-a288-bede67435380"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:17:18.006329 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.006285 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b979de36-45fa-419e-a288-bede67435380-istio-token" (OuterVolumeSpecName: "istio-token") pod "b979de36-45fa-419e-a288-bede67435380" (UID: "b979de36-45fa-419e-a288-bede67435380"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:17:18.006329 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.006310 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b979de36-45fa-419e-a288-bede67435380-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b979de36-45fa-419e-a288-bede67435380" (UID: "b979de36-45fa-419e-a288-bede67435380"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:17:18.006454 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.006324 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b979de36-45fa-419e-a288-bede67435380-kube-api-access-v26tc" (OuterVolumeSpecName: "kube-api-access-v26tc") pod "b979de36-45fa-419e-a288-bede67435380" (UID: "b979de36-45fa-419e-a288-bede67435380"). InnerVolumeSpecName "kube-api-access-v26tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:17:18.105448 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.105410 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-credential-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:17:18.105448 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.105443 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v26tc\" (UniqueName: \"kubernetes.io/projected/b979de36-45fa-419e-a288-bede67435380-kube-api-access-v26tc\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:17:18.105448 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.105456 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b979de36-45fa-419e-a288-bede67435380-istiod-ca-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:17:18.105705 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.105466 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b979de36-45fa-419e-a288-bede67435380-istio-token\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:17:18.105705 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.105475 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b979de36-45fa-419e-a288-bede67435380-istio-podinfo\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:17:18.105705 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.105482 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-istio-data\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:17:18.105705 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.105490 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b979de36-45fa-419e-a288-bede67435380-istio-envoy\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:17:18.812563 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.812506 2585 generic.go:356] "Generic (PLEG): container finished" podID="b979de36-45fa-419e-a288-bede67435380" containerID="d75b0437ff1a870e3e28376a73f862937234e4a4fc0e33eab769e3de8f7df215" exitCode=0 Jul 10 17:17:18.812829 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.812614 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" Jul 10 17:17:18.812829 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.812606 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" event={"ID":"b979de36-45fa-419e-a288-bede67435380","Type":"ContainerDied","Data":"d75b0437ff1a870e3e28376a73f862937234e4a4fc0e33eab769e3de8f7df215"} Jul 10 17:17:18.812829 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.812777 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn" event={"ID":"b979de36-45fa-419e-a288-bede67435380","Type":"ContainerDied","Data":"f8e4c50fae00d4e328edb6ab921c25ea02eb8c908e18733e489cfa6f29e423c4"} Jul 10 17:17:18.812829 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.812818 2585 scope.go:117] "RemoveContainer" containerID="d75b0437ff1a870e3e28376a73f862937234e4a4fc0e33eab769e3de8f7df215" Jul 10 17:17:18.823784 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.823631 2585 scope.go:117] "RemoveContainer" containerID="111d768bafb4d93c12b6e4cef36766c240738a8dc1676013c404cc142f5ecc3e" Jul 10 17:17:18.835382 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.835356 2585 scope.go:117] "RemoveContainer" containerID="d75b0437ff1a870e3e28376a73f862937234e4a4fc0e33eab769e3de8f7df215" Jul 10 17:17:18.835782 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:17:18.835738 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d75b0437ff1a870e3e28376a73f862937234e4a4fc0e33eab769e3de8f7df215\": container with ID starting with d75b0437ff1a870e3e28376a73f862937234e4a4fc0e33eab769e3de8f7df215 not found: ID does not exist" containerID="d75b0437ff1a870e3e28376a73f862937234e4a4fc0e33eab769e3de8f7df215" Jul 10 17:17:18.835883 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.835794 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d75b0437ff1a870e3e28376a73f862937234e4a4fc0e33eab769e3de8f7df215"} err="failed to get container status \"d75b0437ff1a870e3e28376a73f862937234e4a4fc0e33eab769e3de8f7df215\": rpc error: code = NotFound desc = could not find container \"d75b0437ff1a870e3e28376a73f862937234e4a4fc0e33eab769e3de8f7df215\": container with ID starting with d75b0437ff1a870e3e28376a73f862937234e4a4fc0e33eab769e3de8f7df215 not found: ID does not exist" Jul 10 17:17:18.835883 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.835814 2585 scope.go:117] "RemoveContainer" containerID="111d768bafb4d93c12b6e4cef36766c240738a8dc1676013c404cc142f5ecc3e" Jul 10 17:17:18.836145 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:17:18.836124 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111d768bafb4d93c12b6e4cef36766c240738a8dc1676013c404cc142f5ecc3e\": container with ID starting with 111d768bafb4d93c12b6e4cef36766c240738a8dc1676013c404cc142f5ecc3e not found: ID does not exist" containerID="111d768bafb4d93c12b6e4cef36766c240738a8dc1676013c404cc142f5ecc3e" Jul 10 17:17:18.836211 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.836155 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111d768bafb4d93c12b6e4cef36766c240738a8dc1676013c404cc142f5ecc3e"} err="failed to get container status \"111d768bafb4d93c12b6e4cef36766c240738a8dc1676013c404cc142f5ecc3e\": rpc error: code = NotFound desc = could not find container \"111d768bafb4d93c12b6e4cef36766c240738a8dc1676013c404cc142f5ecc3e\": container with ID starting with 111d768bafb4d93c12b6e4cef36766c240738a8dc1676013c404cc142f5ecc3e not found: ID does not exist" Jul 10 17:17:18.841394 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.841356 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn"] Jul 10 17:17:18.853330 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:18.853279 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a0f0a2-openshift-default-57c87675c9-2j6mn"] Jul 10 17:17:19.326405 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:19.326368 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b979de36-45fa-419e-a288-bede67435380" path="/var/lib/kubelet/pods/b979de36-45fa-419e-a288-bede67435380/volumes" Jul 10 17:17:21.901089 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:21.901045 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:17:21.906169 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:21.906134 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr"] Jul 10 17:17:21.906549 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:21.906535 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b979de36-45fa-419e-a288-bede67435380" containerName="istio-proxy" Jul 10 17:17:21.906625 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:21.906551 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b979de36-45fa-419e-a288-bede67435380" containerName="istio-proxy" Jul 10 17:17:21.906673 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:21.906632 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b979de36-45fa-419e-a288-bede67435380" containerName="istio-proxy" Jul 10 17:17:21.906673 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:21.906641 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b979de36-45fa-419e-a288-bede67435380" containerName="istio-proxy" Jul 10 17:17:21.910568 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:21.910537 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:21.913999 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:21.913973 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dd88b4-openshift-default-dockercfg-xs2fc\"" Jul 10 17:17:21.922782 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:21.922733 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr"] Jul 10 17:17:21.938607 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:21.938563 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-workload-socket\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:21.938832 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:21.938669 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-credential-socket\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:21.938832 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:21.938737 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80414e7b-8c37-4572-9c81-88adce7834ca-istio-token\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:21.938832 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:21.938821 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-istio-envoy\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:21.939006 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:21.938894 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80414e7b-8c37-4572-9c81-88adce7834ca-istiod-ca-cert\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:21.939006 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:21.938937 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-istio-data\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:21.939006 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:21.938962 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svtsg\" (UniqueName: \"kubernetes.io/projected/80414e7b-8c37-4572-9c81-88adce7834ca-kube-api-access-svtsg\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:21.939124 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:21.939052 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80414e7b-8c37-4572-9c81-88adce7834ca-istio-podinfo\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:21.939124 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:21.939113 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-workload-certs\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.039661 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.039628 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-credential-socket\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.039905 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.039681 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80414e7b-8c37-4572-9c81-88adce7834ca-istio-token\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.039905 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.039717 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-istio-envoy\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.039905 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.039781 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80414e7b-8c37-4572-9c81-88adce7834ca-istiod-ca-cert\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.039905 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.039822 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-istio-data\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.039905 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.039845 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svtsg\" (UniqueName: \"kubernetes.io/projected/80414e7b-8c37-4572-9c81-88adce7834ca-kube-api-access-svtsg\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.040220 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.040142 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-credential-socket\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.040220 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.040152 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80414e7b-8c37-4572-9c81-88adce7834ca-istio-podinfo\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.040329 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.040252 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-workload-certs\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.040329 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.040259 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-istio-data\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.040329 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.040288 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-workload-socket\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.040621 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.040599 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-workload-certs\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.040714 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.040645 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-workload-socket\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.041074 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.041039 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80414e7b-8c37-4572-9c81-88adce7834ca-istiod-ca-cert\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.042416 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.042392 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-istio-envoy\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.042672 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.042652 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80414e7b-8c37-4572-9c81-88adce7834ca-istio-podinfo\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.050006 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.049968 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80414e7b-8c37-4572-9c81-88adce7834ca-istio-token\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.050142 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.050031 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svtsg\" (UniqueName: \"kubernetes.io/projected/80414e7b-8c37-4572-9c81-88adce7834ca-kube-api-access-svtsg\") pod \"e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.223606 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.223497 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:22.383197 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.383164 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr"] Jul 10 17:17:22.385092 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:17:22.385059 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80414e7b_8c37_4572_9c81_88adce7834ca.slice/crio-8aa6307d357bb8a210432677d4e349c6691bf50891b8435a09b106eda6a35c20 WatchSource:0}: Error finding container 8aa6307d357bb8a210432677d4e349c6691bf50891b8435a09b106eda6a35c20: Status 404 returned error can't find the container with id 8aa6307d357bb8a210432677d4e349c6691bf50891b8435a09b106eda6a35c20 Jul 10 17:17:22.388080 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.388022 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:17:22.388214 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.388152 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:17:22.388214 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.388190 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:17:22.844048 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.843995 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" event={"ID":"80414e7b-8c37-4572-9c81-88adce7834ca","Type":"ContainerStarted","Data":"f5629ffd570e6530837ba1710a1301fe493ce66f2aa34af83b85b2c962926364"} Jul 10 17:17:22.844284 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.844154 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" event={"ID":"80414e7b-8c37-4572-9c81-88adce7834ca","Type":"ContainerStarted","Data":"8aa6307d357bb8a210432677d4e349c6691bf50891b8435a09b106eda6a35c20"} Jul 10 17:17:22.875534 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:22.875458 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" podStartSLOduration=1.875435982 podStartE2EDuration="1.875435982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:17:22.870939847 +0000 UTC m=+2462.292816342" watchObservedRunningTime="2026-07-10 17:17:22.875435982 +0000 UTC m=+2462.297312454" Jul 10 17:17:23.223948 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:23.223905 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:23.229410 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:23.229380 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:23.854903 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:23.854861 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:23.859028 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:23.858768 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:30.907035 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:30.906997 2585 generic.go:356] "Generic (PLEG): container finished" podID="76420a4e-6c81-4db7-bced-57e7237fd7b2" containerID="5cc062d6293842020f1b0fe6c4cf95eddf0f257fdab84c30e5ade66582c65bda" exitCode=0 Jul 10 17:17:30.907496 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:30.907071 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" event={"ID":"76420a4e-6c81-4db7-bced-57e7237fd7b2","Type":"ContainerDied","Data":"5cc062d6293842020f1b0fe6c4cf95eddf0f257fdab84c30e5ade66582c65bda"} Jul 10 17:17:30.907540 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:30.907516 2585 scope.go:117] "RemoveContainer" containerID="5cc062d6293842020f1b0fe6c4cf95eddf0f257fdab84c30e5ade66582c65bda" Jul 10 17:17:30.908703 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:30.908670 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:17:30.908813 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:30.908762 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:17:30.908813 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:30.908798 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:17:31.799899 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:31.799853 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:31.915171 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:31.915130 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" event={"ID":"76420a4e-6c81-4db7-bced-57e7237fd7b2","Type":"ContainerStarted","Data":"6e02bfad68abd062d2e86d9858f1a780171d0640286f8b1ca01ffb993fe28830"} Jul 10 17:17:32.712967 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:32.712919 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:17:32.898963 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:32.898915 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:32.900452 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:32.900422 2585 patch_prober.go:28] interesting pod/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 17:17:32.900625 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:32.900499 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" podUID="76420a4e-6c81-4db7-bced-57e7237fd7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 17:17:33.899205 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:33.899154 2585 patch_prober.go:28] interesting pod/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 17:17:33.899646 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:33.899228 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" podUID="76420a4e-6c81-4db7-bced-57e7237fd7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 17:17:34.898907 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:34.898854 2585 patch_prober.go:28] interesting pod/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 17:17:34.899156 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:34.898948 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" podUID="76420a4e-6c81-4db7-bced-57e7237fd7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 17:17:35.898961 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:35.898915 2585 patch_prober.go:28] interesting pod/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 17:17:35.899417 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:35.898979 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" podUID="76420a4e-6c81-4db7-bced-57e7237fd7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 17:17:36.903005 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:36.902970 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:36.903443 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:36.903260 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:36.904027 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:36.904004 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:17:39.894988 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:39.894935 2585 patch_prober.go:28] interesting pod/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 17:17:39.895388 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:39.895009 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" podUID="80414e7b-8c37-4572-9c81-88adce7834ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 17:17:40.985513 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:40.985481 2585 generic.go:356] "Generic (PLEG): container finished" podID="80414e7b-8c37-4572-9c81-88adce7834ca" containerID="f5629ffd570e6530837ba1710a1301fe493ce66f2aa34af83b85b2c962926364" exitCode=0 Jul 10 17:17:40.985991 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:40.985555 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" event={"ID":"80414e7b-8c37-4572-9c81-88adce7834ca","Type":"ContainerDied","Data":"f5629ffd570e6530837ba1710a1301fe493ce66f2aa34af83b85b2c962926364"} Jul 10 17:17:40.986073 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:40.986011 2585 scope.go:117] "RemoveContainer" containerID="f5629ffd570e6530837ba1710a1301fe493ce66f2aa34af83b85b2c962926364" Jul 10 17:17:40.987099 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:40.987068 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:17:40.987205 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:40.987157 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:17:40.987265 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:40.987208 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:17:41.993403 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:41.993365 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" event={"ID":"80414e7b-8c37-4572-9c81-88adce7834ca","Type":"ContainerStarted","Data":"cc876241f61d0f3c6160825b109fe150715e00066936586f532bedd96f1666f0"} Jul 10 17:17:42.223767 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:42.223694 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:42.223987 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:42.223962 2585 patch_prober.go:28] interesting pod/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 10 17:17:42.224038 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:42.224018 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" podUID="80414e7b-8c37-4572-9c81-88adce7834ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 10 17:17:43.224583 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:43.224537 2585 patch_prober.go:28] interesting pod/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 10 17:17:43.225038 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:43.224614 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" podUID="80414e7b-8c37-4572-9c81-88adce7834ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 10 17:17:44.224836 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:44.224795 2585 patch_prober.go:28] interesting pod/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 10 17:17:44.225347 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:44.224859 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" podUID="80414e7b-8c37-4572-9c81-88adce7834ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 10 17:17:45.224422 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:45.224381 2585 patch_prober.go:28] interesting pod/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 10 17:17:45.224616 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:45.224439 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" podUID="80414e7b-8c37-4572-9c81-88adce7834ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 10 17:17:46.227825 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:46.227789 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:46.228350 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:46.228206 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:46.228974 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:46.228948 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:17:51.203058 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:17:51.203019 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:18:02.848957 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:02.848923 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb"] Jul 10 17:18:02.849388 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:02.849139 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" podUID="76420a4e-6c81-4db7-bced-57e7237fd7b2" containerName="istio-proxy" containerID="cri-o://6e02bfad68abd062d2e86d9858f1a780171d0640286f8b1ca01ffb993fe28830" gracePeriod=30 Jul 10 17:18:06.905808 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:06.905773 2585 patch_prober.go:28] interesting pod/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 17:18:06.906193 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:06.905828 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" podUID="76420a4e-6c81-4db7-bced-57e7237fd7b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 17:18:08.107668 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.107641 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:18:08.146061 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.145967 2585 generic.go:356] "Generic (PLEG): container finished" podID="76420a4e-6c81-4db7-bced-57e7237fd7b2" containerID="6e02bfad68abd062d2e86d9858f1a780171d0640286f8b1ca01ffb993fe28830" exitCode=0 Jul 10 17:18:08.146061 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.146030 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" Jul 10 17:18:08.146259 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.146051 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" event={"ID":"76420a4e-6c81-4db7-bced-57e7237fd7b2","Type":"ContainerDied","Data":"6e02bfad68abd062d2e86d9858f1a780171d0640286f8b1ca01ffb993fe28830"} Jul 10 17:18:08.146259 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.146144 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb" event={"ID":"76420a4e-6c81-4db7-bced-57e7237fd7b2","Type":"ContainerDied","Data":"d78b7ab05e05e6f90138f43625ec5ec7ef5e2f1d3dba549eceb0a05940bdd80c"} Jul 10 17:18:08.146259 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.146173 2585 scope.go:117] "RemoveContainer" containerID="6e02bfad68abd062d2e86d9858f1a780171d0640286f8b1ca01ffb993fe28830" Jul 10 17:18:08.155848 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.155830 2585 scope.go:117] "RemoveContainer" containerID="5cc062d6293842020f1b0fe6c4cf95eddf0f257fdab84c30e5ade66582c65bda" Jul 10 17:18:08.163998 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.163981 2585 scope.go:117] "RemoveContainer" containerID="6e02bfad68abd062d2e86d9858f1a780171d0640286f8b1ca01ffb993fe28830" Jul 10 17:18:08.164287 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:18:08.164268 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e02bfad68abd062d2e86d9858f1a780171d0640286f8b1ca01ffb993fe28830\": container with ID starting with 6e02bfad68abd062d2e86d9858f1a780171d0640286f8b1ca01ffb993fe28830 not found: ID does not exist" containerID="6e02bfad68abd062d2e86d9858f1a780171d0640286f8b1ca01ffb993fe28830" Jul 10 17:18:08.164341 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.164296 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e02bfad68abd062d2e86d9858f1a780171d0640286f8b1ca01ffb993fe28830"} err="failed to get container status \"6e02bfad68abd062d2e86d9858f1a780171d0640286f8b1ca01ffb993fe28830\": rpc error: code = NotFound desc = could not find container \"6e02bfad68abd062d2e86d9858f1a780171d0640286f8b1ca01ffb993fe28830\": container with ID starting with 6e02bfad68abd062d2e86d9858f1a780171d0640286f8b1ca01ffb993fe28830 not found: ID does not exist" Jul 10 17:18:08.164341 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.164315 2585 scope.go:117] "RemoveContainer" containerID="5cc062d6293842020f1b0fe6c4cf95eddf0f257fdab84c30e5ade66582c65bda" Jul 10 17:18:08.164553 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:18:08.164534 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc062d6293842020f1b0fe6c4cf95eddf0f257fdab84c30e5ade66582c65bda\": container with ID starting with 5cc062d6293842020f1b0fe6c4cf95eddf0f257fdab84c30e5ade66582c65bda not found: ID does not exist" containerID="5cc062d6293842020f1b0fe6c4cf95eddf0f257fdab84c30e5ade66582c65bda" Jul 10 17:18:08.164601 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.164556 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc062d6293842020f1b0fe6c4cf95eddf0f257fdab84c30e5ade66582c65bda"} err="failed to get container status \"5cc062d6293842020f1b0fe6c4cf95eddf0f257fdab84c30e5ade66582c65bda\": rpc error: code = NotFound desc = could not find container \"5cc062d6293842020f1b0fe6c4cf95eddf0f257fdab84c30e5ade66582c65bda\": container with ID starting with 5cc062d6293842020f1b0fe6c4cf95eddf0f257fdab84c30e5ade66582c65bda not found: ID does not exist" Jul 10 17:18:08.218070 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.218037 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-data\") pod \"76420a4e-6c81-4db7-bced-57e7237fd7b2\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " Jul 10 17:18:08.218218 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.218077 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttgvc\" (UniqueName: \"kubernetes.io/projected/76420a4e-6c81-4db7-bced-57e7237fd7b2-kube-api-access-ttgvc\") pod \"76420a4e-6c81-4db7-bced-57e7237fd7b2\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " Jul 10 17:18:08.218218 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.218102 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-workload-socket\") pod \"76420a4e-6c81-4db7-bced-57e7237fd7b2\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " Jul 10 17:18:08.218218 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.218122 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-credential-socket\") pod \"76420a4e-6c81-4db7-bced-57e7237fd7b2\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " Jul 10 17:18:08.218218 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.218147 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-envoy\") pod \"76420a4e-6c81-4db7-bced-57e7237fd7b2\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " Jul 10 17:18:08.218424 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.218273 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76420a4e-6c81-4db7-bced-57e7237fd7b2-istiod-ca-cert\") pod \"76420a4e-6c81-4db7-bced-57e7237fd7b2\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " Jul 10 17:18:08.218424 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.218308 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-workload-certs\") pod \"76420a4e-6c81-4db7-bced-57e7237fd7b2\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " Jul 10 17:18:08.218424 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.218360 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-podinfo\") pod \"76420a4e-6c81-4db7-bced-57e7237fd7b2\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " Jul 10 17:18:08.218424 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.218374 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "76420a4e-6c81-4db7-bced-57e7237fd7b2" (UID: "76420a4e-6c81-4db7-bced-57e7237fd7b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:18:08.218424 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.218412 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-token\") pod \"76420a4e-6c81-4db7-bced-57e7237fd7b2\" (UID: \"76420a4e-6c81-4db7-bced-57e7237fd7b2\") " Jul 10 17:18:08.218678 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.218480 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "76420a4e-6c81-4db7-bced-57e7237fd7b2" (UID: "76420a4e-6c81-4db7-bced-57e7237fd7b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:18:08.218678 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.218560 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "76420a4e-6c81-4db7-bced-57e7237fd7b2" (UID: "76420a4e-6c81-4db7-bced-57e7237fd7b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:18:08.218887 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.218791 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-workload-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:18:08.218887 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.218812 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-credential-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:18:08.218887 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.218828 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-workload-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:18:08.219046 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.219011 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-data" (OuterVolumeSpecName: "istio-data") pod "76420a4e-6c81-4db7-bced-57e7237fd7b2" (UID: "76420a4e-6c81-4db7-bced-57e7237fd7b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:18:08.219108 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.219041 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76420a4e-6c81-4db7-bced-57e7237fd7b2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "76420a4e-6c81-4db7-bced-57e7237fd7b2" (UID: "76420a4e-6c81-4db7-bced-57e7237fd7b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:18:08.220538 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.220512 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "76420a4e-6c81-4db7-bced-57e7237fd7b2" (UID: "76420a4e-6c81-4db7-bced-57e7237fd7b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:18:08.220649 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.220553 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-token" (OuterVolumeSpecName: "istio-token") pod "76420a4e-6c81-4db7-bced-57e7237fd7b2" (UID: "76420a4e-6c81-4db7-bced-57e7237fd7b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:18:08.220738 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.220718 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "76420a4e-6c81-4db7-bced-57e7237fd7b2" (UID: "76420a4e-6c81-4db7-bced-57e7237fd7b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:18:08.220871 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.220818 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76420a4e-6c81-4db7-bced-57e7237fd7b2-kube-api-access-ttgvc" (OuterVolumeSpecName: "kube-api-access-ttgvc") pod "76420a4e-6c81-4db7-bced-57e7237fd7b2" (UID: "76420a4e-6c81-4db7-bced-57e7237fd7b2"). InnerVolumeSpecName "kube-api-access-ttgvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:18:08.320342 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.320311 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76420a4e-6c81-4db7-bced-57e7237fd7b2-istiod-ca-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:18:08.320494 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.320359 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-podinfo\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:18:08.320494 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.320400 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-token\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:18:08.320494 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.320433 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-data\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:18:08.320494 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.320449 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttgvc\" (UniqueName: \"kubernetes.io/projected/76420a4e-6c81-4db7-bced-57e7237fd7b2-kube-api-access-ttgvc\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:18:08.320494 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.320470 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76420a4e-6c81-4db7-bced-57e7237fd7b2-istio-envoy\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:18:08.470761 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.470715 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb"] Jul 10 17:18:08.475879 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:08.475855 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5e0f6f-openshift-default-cf5569b67-bndtb"] Jul 10 17:18:09.326325 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:09.326292 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76420a4e-6c81-4db7-bced-57e7237fd7b2" path="/var/lib/kubelet/pods/76420a4e-6c81-4db7-bced-57e7237fd7b2/volumes" Jul 10 17:18:16.807352 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:16.807263 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:18:28.935054 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:28.935015 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr"] Jul 10 17:18:28.935486 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:28.935243 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" podUID="80414e7b-8c37-4572-9c81-88adce7834ca" containerName="istio-proxy" containerID="cri-o://cc876241f61d0f3c6160825b109fe150715e00066936586f532bedd96f1666f0" gracePeriod=30 Jul 10 17:18:29.810103 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.810068 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr"] Jul 10 17:18:29.810442 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.810429 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76420a4e-6c81-4db7-bced-57e7237fd7b2" containerName="istio-proxy" Jul 10 17:18:29.810497 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.810445 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="76420a4e-6c81-4db7-bced-57e7237fd7b2" containerName="istio-proxy" Jul 10 17:18:29.810497 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.810458 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76420a4e-6c81-4db7-bced-57e7237fd7b2" containerName="istio-proxy" Jul 10 17:18:29.810497 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.810464 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="76420a4e-6c81-4db7-bced-57e7237fd7b2" containerName="istio-proxy" Jul 10 17:18:29.810497 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.810476 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b979de36-45fa-419e-a288-bede67435380" containerName="istio-proxy" Jul 10 17:18:29.810497 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.810482 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b979de36-45fa-419e-a288-bede67435380" containerName="istio-proxy" Jul 10 17:18:29.810665 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.810565 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="76420a4e-6c81-4db7-bced-57e7237fd7b2" containerName="istio-proxy" Jul 10 17:18:29.812879 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.812860 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:29.815765 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.815720 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-fd7cfb-openshift-default-dockercfg-4phjw\"" Jul 10 17:18:29.826285 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.826259 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr"] Jul 10 17:18:29.936659 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.936622 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-workload-certs\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:29.937066 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.936664 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-envoy\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:29.937066 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.936754 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-token\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:29.937066 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.936795 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-credential-socket\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:29.937066 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.936832 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-workload-socket\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:29.937066 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.936889 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-podinfo\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:29.937066 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.936941 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e2b632-14ce-43c9-89d8-7daae3e77154-istiod-ca-cert\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:29.937066 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.936959 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb6xb\" (UniqueName: \"kubernetes.io/projected/07e2b632-14ce-43c9-89d8-7daae3e77154-kube-api-access-wb6xb\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:29.937066 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:29.936985 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-data\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.038218 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.038178 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-workload-certs\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.038218 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.038219 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-envoy\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.038452 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.038256 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-token\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.038452 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.038273 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-credential-socket\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.038452 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.038304 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-workload-socket\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.038452 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.038346 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-podinfo\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.038452 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.038409 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e2b632-14ce-43c9-89d8-7daae3e77154-istiod-ca-cert\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.038452 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.038445 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb6xb\" (UniqueName: \"kubernetes.io/projected/07e2b632-14ce-43c9-89d8-7daae3e77154-kube-api-access-wb6xb\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.038716 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.038484 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-data\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.038716 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.038626 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-workload-certs\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.038716 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.038668 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-credential-socket\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.039092 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.039055 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-workload-socket\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.039092 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.039068 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-data\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.039296 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.039273 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e2b632-14ce-43c9-89d8-7daae3e77154-istiod-ca-cert\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.040821 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.040765 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-envoy\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.041045 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.041030 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-podinfo\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.050122 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.050091 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-token\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.050243 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.050146 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb6xb\" (UniqueName: \"kubernetes.io/projected/07e2b632-14ce-43c9-89d8-7daae3e77154-kube-api-access-wb6xb\") pod \"e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.123339 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.123302 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:30.267864 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.267837 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr"] Jul 10 17:18:30.269254 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:18:30.269228 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e2b632_14ce_43c9_89d8_7daae3e77154.slice/crio-5c7fd472850be0c3fd3289db4598295ea7224c417566ae5bc096c10a879e85ba WatchSource:0}: Error finding container 5c7fd472850be0c3fd3289db4598295ea7224c417566ae5bc096c10a879e85ba: Status 404 returned error can't find the container with id 5c7fd472850be0c3fd3289db4598295ea7224c417566ae5bc096c10a879e85ba Jul 10 17:18:30.271538 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.271503 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:18:30.271633 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.271578 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:18:30.271675 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.271648 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:18:30.276358 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:30.276326 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" event={"ID":"07e2b632-14ce-43c9-89d8-7daae3e77154","Type":"ContainerStarted","Data":"5c7fd472850be0c3fd3289db4598295ea7224c417566ae5bc096c10a879e85ba"} Jul 10 17:18:31.230984 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:31.230941 2585 patch_prober.go:28] interesting pod/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 10 17:18:31.231379 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:31.230997 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" podUID="80414e7b-8c37-4572-9c81-88adce7834ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 17:18:31.283712 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:31.283675 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" event={"ID":"07e2b632-14ce-43c9-89d8-7daae3e77154","Type":"ContainerStarted","Data":"5d1f35696ac7dbb9ed3a12d30748d893cbbdbdbd6f52d7389411a2226242b4a1"} Jul 10 17:18:31.307830 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:31.307777 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" podStartSLOduration=2.307760782 podStartE2EDuration="2.307760782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:18:31.304533937 +0000 UTC m=+2530.726410447" watchObservedRunningTime="2026-07-10 17:18:31.307760782 +0000 UTC m=+2530.729637239" Jul 10 17:18:32.123524 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:32.123486 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:32.128724 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:32.128697 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:32.289207 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:32.289169 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:32.290135 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:32.290115 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:34.193618 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.193595 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:18:34.279998 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.279921 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-credential-socket\") pod \"80414e7b-8c37-4572-9c81-88adce7834ca\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " Jul 10 17:18:34.279998 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.279959 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svtsg\" (UniqueName: \"kubernetes.io/projected/80414e7b-8c37-4572-9c81-88adce7834ca-kube-api-access-svtsg\") pod \"80414e7b-8c37-4572-9c81-88adce7834ca\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " Jul 10 17:18:34.279998 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.279979 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80414e7b-8c37-4572-9c81-88adce7834ca-istio-podinfo\") pod \"80414e7b-8c37-4572-9c81-88adce7834ca\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " Jul 10 17:18:34.280304 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.280069 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80414e7b-8c37-4572-9c81-88adce7834ca-istio-token\") pod \"80414e7b-8c37-4572-9c81-88adce7834ca\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " Jul 10 17:18:34.280304 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.280102 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80414e7b-8c37-4572-9c81-88adce7834ca-istiod-ca-cert\") pod \"80414e7b-8c37-4572-9c81-88adce7834ca\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " Jul 10 17:18:34.280304 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.280128 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-istio-envoy\") pod \"80414e7b-8c37-4572-9c81-88adce7834ca\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " Jul 10 17:18:34.280304 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.280213 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "80414e7b-8c37-4572-9c81-88adce7834ca" (UID: "80414e7b-8c37-4572-9c81-88adce7834ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:18:34.280304 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.280233 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-workload-certs\") pod \"80414e7b-8c37-4572-9c81-88adce7834ca\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " Jul 10 17:18:34.280304 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.280271 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-workload-socket\") pod \"80414e7b-8c37-4572-9c81-88adce7834ca\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " Jul 10 17:18:34.280304 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.280303 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-istio-data\") pod \"80414e7b-8c37-4572-9c81-88adce7834ca\" (UID: \"80414e7b-8c37-4572-9c81-88adce7834ca\") " Jul 10 17:18:34.280657 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.280548 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "80414e7b-8c37-4572-9c81-88adce7834ca" (UID: "80414e7b-8c37-4572-9c81-88adce7834ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:18:34.280657 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.280562 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80414e7b-8c37-4572-9c81-88adce7834ca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "80414e7b-8c37-4572-9c81-88adce7834ca" (UID: "80414e7b-8c37-4572-9c81-88adce7834ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:18:34.280657 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.280565 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "80414e7b-8c37-4572-9c81-88adce7834ca" (UID: "80414e7b-8c37-4572-9c81-88adce7834ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:18:34.280657 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.280647 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-workload-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:18:34.280934 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.280666 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-workload-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:18:34.280934 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.280679 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-credential-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:18:34.280934 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.280694 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80414e7b-8c37-4572-9c81-88adce7834ca-istiod-ca-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:18:34.281445 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.281419 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-istio-data" (OuterVolumeSpecName: "istio-data") pod "80414e7b-8c37-4572-9c81-88adce7834ca" (UID: "80414e7b-8c37-4572-9c81-88adce7834ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:18:34.282330 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.282302 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "80414e7b-8c37-4572-9c81-88adce7834ca" (UID: "80414e7b-8c37-4572-9c81-88adce7834ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:18:34.282440 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.282358 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80414e7b-8c37-4572-9c81-88adce7834ca-istio-token" (OuterVolumeSpecName: "istio-token") pod "80414e7b-8c37-4572-9c81-88adce7834ca" (UID: "80414e7b-8c37-4572-9c81-88adce7834ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:18:34.282440 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.282411 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80414e7b-8c37-4572-9c81-88adce7834ca-kube-api-access-svtsg" (OuterVolumeSpecName: "kube-api-access-svtsg") pod "80414e7b-8c37-4572-9c81-88adce7834ca" (UID: "80414e7b-8c37-4572-9c81-88adce7834ca"). InnerVolumeSpecName "kube-api-access-svtsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:18:34.282673 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.282659 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80414e7b-8c37-4572-9c81-88adce7834ca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "80414e7b-8c37-4572-9c81-88adce7834ca" (UID: "80414e7b-8c37-4572-9c81-88adce7834ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:18:34.300625 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.300598 2585 generic.go:356] "Generic (PLEG): container finished" podID="80414e7b-8c37-4572-9c81-88adce7834ca" containerID="cc876241f61d0f3c6160825b109fe150715e00066936586f532bedd96f1666f0" exitCode=0 Jul 10 17:18:34.300704 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.300638 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" Jul 10 17:18:34.300704 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.300674 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" event={"ID":"80414e7b-8c37-4572-9c81-88adce7834ca","Type":"ContainerDied","Data":"cc876241f61d0f3c6160825b109fe150715e00066936586f532bedd96f1666f0"} Jul 10 17:18:34.300841 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.300729 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr" event={"ID":"80414e7b-8c37-4572-9c81-88adce7834ca","Type":"ContainerDied","Data":"8aa6307d357bb8a210432677d4e349c6691bf50891b8435a09b106eda6a35c20"} Jul 10 17:18:34.300841 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.300781 2585 scope.go:117] "RemoveContainer" containerID="cc876241f61d0f3c6160825b109fe150715e00066936586f532bedd96f1666f0" Jul 10 17:18:34.310293 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.310274 2585 scope.go:117] "RemoveContainer" containerID="f5629ffd570e6530837ba1710a1301fe493ce66f2aa34af83b85b2c962926364" Jul 10 17:18:34.318060 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.318043 2585 scope.go:117] "RemoveContainer" containerID="cc876241f61d0f3c6160825b109fe150715e00066936586f532bedd96f1666f0" Jul 10 17:18:34.318300 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:18:34.318283 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc876241f61d0f3c6160825b109fe150715e00066936586f532bedd96f1666f0\": container with ID starting with cc876241f61d0f3c6160825b109fe150715e00066936586f532bedd96f1666f0 not found: ID does not exist" containerID="cc876241f61d0f3c6160825b109fe150715e00066936586f532bedd96f1666f0" Jul 10 17:18:34.318368 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.318313 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc876241f61d0f3c6160825b109fe150715e00066936586f532bedd96f1666f0"} err="failed to get container status \"cc876241f61d0f3c6160825b109fe150715e00066936586f532bedd96f1666f0\": rpc error: code = NotFound desc = could not find container \"cc876241f61d0f3c6160825b109fe150715e00066936586f532bedd96f1666f0\": container with ID starting with cc876241f61d0f3c6160825b109fe150715e00066936586f532bedd96f1666f0 not found: ID does not exist" Jul 10 17:18:34.318368 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.318336 2585 scope.go:117] "RemoveContainer" containerID="f5629ffd570e6530837ba1710a1301fe493ce66f2aa34af83b85b2c962926364" Jul 10 17:18:34.318561 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:18:34.318541 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5629ffd570e6530837ba1710a1301fe493ce66f2aa34af83b85b2c962926364\": container with ID starting with f5629ffd570e6530837ba1710a1301fe493ce66f2aa34af83b85b2c962926364 not found: ID does not exist" containerID="f5629ffd570e6530837ba1710a1301fe493ce66f2aa34af83b85b2c962926364" Jul 10 17:18:34.318602 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.318568 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5629ffd570e6530837ba1710a1301fe493ce66f2aa34af83b85b2c962926364"} err="failed to get container status \"f5629ffd570e6530837ba1710a1301fe493ce66f2aa34af83b85b2c962926364\": rpc error: code = NotFound desc = could not find container \"f5629ffd570e6530837ba1710a1301fe493ce66f2aa34af83b85b2c962926364\": container with ID starting with f5629ffd570e6530837ba1710a1301fe493ce66f2aa34af83b85b2c962926364 not found: ID does not exist" Jul 10 17:18:34.327029 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.326998 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr"] Jul 10 17:18:34.330960 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.330942 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dd88b4-openshift-default-58798b54d5-7wdpr"] Jul 10 17:18:34.381408 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.381260 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svtsg\" (UniqueName: \"kubernetes.io/projected/80414e7b-8c37-4572-9c81-88adce7834ca-kube-api-access-svtsg\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:18:34.381408 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.381291 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80414e7b-8c37-4572-9c81-88adce7834ca-istio-podinfo\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:18:34.381408 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.381307 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80414e7b-8c37-4572-9c81-88adce7834ca-istio-token\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:18:34.381408 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.381321 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-istio-envoy\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:18:34.381408 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:34.381335 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80414e7b-8c37-4572-9c81-88adce7834ca-istio-data\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:18:35.326170 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:35.326139 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80414e7b-8c37-4572-9c81-88adce7834ca" path="/var/lib/kubelet/pods/80414e7b-8c37-4572-9c81-88adce7834ca/volumes" Jul 10 17:18:37.497524 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:37.497487 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:18:38.274338 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.274306 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp"] Jul 10 17:18:38.274886 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.274869 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80414e7b-8c37-4572-9c81-88adce7834ca" containerName="istio-proxy" Jul 10 17:18:38.274935 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.274890 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="80414e7b-8c37-4572-9c81-88adce7834ca" containerName="istio-proxy" Jul 10 17:18:38.274935 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.274901 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80414e7b-8c37-4572-9c81-88adce7834ca" containerName="istio-proxy" Jul 10 17:18:38.274935 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.274910 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="80414e7b-8c37-4572-9c81-88adce7834ca" containerName="istio-proxy" Jul 10 17:18:38.275025 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.275017 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="80414e7b-8c37-4572-9c81-88adce7834ca" containerName="istio-proxy" Jul 10 17:18:38.275061 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.275030 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="76420a4e-6c81-4db7-bced-57e7237fd7b2" containerName="istio-proxy" Jul 10 17:18:38.275061 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.275044 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="80414e7b-8c37-4572-9c81-88adce7834ca" containerName="istio-proxy" Jul 10 17:18:38.278891 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.278875 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.282674 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.282657 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-c9789a-openshift-default-dockercfg-bl9sd\"" Jul 10 17:18:38.299028 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.298995 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp"] Jul 10 17:18:38.318958 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.318927 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-data\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.318958 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.318958 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-podinfo\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.319121 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.318982 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-credential-socket\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.319121 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.319088 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-envoy\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.319198 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.319139 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-token\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.319235 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.319201 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md5cf\" (UniqueName: \"kubernetes.io/projected/c35dbe24-690a-4c79-a4da-007f9a351f09-kube-api-access-md5cf\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.319275 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.319235 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-workload-socket\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.319275 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.319251 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35dbe24-690a-4c79-a4da-007f9a351f09-istiod-ca-cert\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.319341 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.319274 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-workload-certs\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.420085 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.420052 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-token\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.420271 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.420096 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md5cf\" (UniqueName: \"kubernetes.io/projected/c35dbe24-690a-4c79-a4da-007f9a351f09-kube-api-access-md5cf\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.420271 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.420122 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-workload-socket\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.420271 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.420243 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35dbe24-690a-4c79-a4da-007f9a351f09-istiod-ca-cert\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.420403 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.420290 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-workload-certs\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.420403 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.420354 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-data\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.420403 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.420384 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-podinfo\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.420552 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.420419 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-credential-socket\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.420552 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.420499 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-envoy\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.420552 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.420504 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-workload-socket\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.420711 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.420658 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-workload-certs\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.420800 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.420776 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-data\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.420859 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.420782 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-credential-socket\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.420951 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.420930 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35dbe24-690a-4c79-a4da-007f9a351f09-istiod-ca-cert\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.422729 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.422707 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-envoy\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.422863 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.422845 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-podinfo\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.429172 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.429146 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md5cf\" (UniqueName: \"kubernetes.io/projected/c35dbe24-690a-4c79-a4da-007f9a351f09-kube-api-access-md5cf\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.429330 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.429313 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-token\") pod \"e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.588873 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.588787 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:38.761170 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.761139 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp"] Jul 10 17:18:38.763650 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:18:38.763615 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35dbe24_690a_4c79_a4da_007f9a351f09.slice/crio-3e44aa5bcbbf4da3d1af564a0ffa65f174af45897e94a729e7b429fe974da05f WatchSource:0}: Error finding container 3e44aa5bcbbf4da3d1af564a0ffa65f174af45897e94a729e7b429fe974da05f: Status 404 returned error can't find the container with id 3e44aa5bcbbf4da3d1af564a0ffa65f174af45897e94a729e7b429fe974da05f Jul 10 17:18:38.765855 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.765811 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:18:38.765950 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.765877 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:18:38.765950 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:38.765905 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:18:39.331614 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:39.331580 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" event={"ID":"c35dbe24-690a-4c79-a4da-007f9a351f09","Type":"ContainerStarted","Data":"0a4ce96218c18a2ebf99f0886fba9fdd168ef5d6fd5647e832b3da4a22a34998"} Jul 10 17:18:39.331811 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:39.331631 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" event={"ID":"c35dbe24-690a-4c79-a4da-007f9a351f09","Type":"ContainerStarted","Data":"3e44aa5bcbbf4da3d1af564a0ffa65f174af45897e94a729e7b429fe974da05f"} Jul 10 17:18:39.353888 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:39.353826 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" podStartSLOduration=1.3537916779999999 podStartE2EDuration="1.353791678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:18:39.352473854 +0000 UTC m=+2538.774350339" watchObservedRunningTime="2026-07-10 17:18:39.353791678 +0000 UTC m=+2538.775668146" Jul 10 17:18:39.589985 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:39.589892 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:39.594759 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:39.594717 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:40.341581 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:40.341516 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:40.346863 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:40.346617 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:45.378387 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:45.378303 2585 generic.go:356] "Generic (PLEG): container finished" podID="07e2b632-14ce-43c9-89d8-7daae3e77154" containerID="5d1f35696ac7dbb9ed3a12d30748d893cbbdbdbd6f52d7389411a2226242b4a1" exitCode=0 Jul 10 17:18:45.378823 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:45.378378 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" event={"ID":"07e2b632-14ce-43c9-89d8-7daae3e77154","Type":"ContainerDied","Data":"5d1f35696ac7dbb9ed3a12d30748d893cbbdbdbd6f52d7389411a2226242b4a1"} Jul 10 17:18:45.378823 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:45.378809 2585 scope.go:117] "RemoveContainer" containerID="5d1f35696ac7dbb9ed3a12d30748d893cbbdbdbd6f52d7389411a2226242b4a1" Jul 10 17:18:45.379802 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:45.379774 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:18:45.379907 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:45.379853 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:18:45.379967 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:45.379903 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:18:46.386097 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:46.386061 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" event={"ID":"07e2b632-14ce-43c9-89d8-7daae3e77154","Type":"ContainerStarted","Data":"16d1416b69a167fe8106ae1432503c5ffd5684d6dffb1bb511142a2205f74aeb"} Jul 10 17:18:47.124559 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:47.124519 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:47.125820 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:47.125791 2585 patch_prober.go:28] interesting pod/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 10 17:18:47.125952 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:47.125851 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" podUID="07e2b632-14ce-43c9-89d8-7daae3e77154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 10 17:18:47.289676 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:47.289637 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:48.124594 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:48.124555 2585 patch_prober.go:28] interesting pod/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 10 17:18:48.125134 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:48.124628 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" podUID="07e2b632-14ce-43c9-89d8-7daae3e77154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 10 17:18:49.103520 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:49.103484 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:18:49.124267 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:49.124236 2585 patch_prober.go:28] interesting pod/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 10 17:18:49.124406 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:49.124285 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" podUID="07e2b632-14ce-43c9-89d8-7daae3e77154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 10 17:18:50.124212 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:50.124139 2585 patch_prober.go:28] interesting pod/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 10 17:18:50.124595 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:50.124235 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" podUID="07e2b632-14ce-43c9-89d8-7daae3e77154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 10 17:18:51.128053 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:51.128021 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:51.129064 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:51.129046 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:18:56.383917 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:56.383884 2585 patch_prober.go:28] interesting pod/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 17:18:56.385220 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:56.383948 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" podUID="c35dbe24-690a-4c79-a4da-007f9a351f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 17:18:57.447883 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:57.447850 2585 generic.go:356] "Generic (PLEG): container finished" podID="c35dbe24-690a-4c79-a4da-007f9a351f09" containerID="0a4ce96218c18a2ebf99f0886fba9fdd168ef5d6fd5647e832b3da4a22a34998" exitCode=0 Jul 10 17:18:57.448322 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:57.447921 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" event={"ID":"c35dbe24-690a-4c79-a4da-007f9a351f09","Type":"ContainerDied","Data":"0a4ce96218c18a2ebf99f0886fba9fdd168ef5d6fd5647e832b3da4a22a34998"} Jul 10 17:18:57.448446 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:57.448429 2585 scope.go:117] "RemoveContainer" containerID="0a4ce96218c18a2ebf99f0886fba9fdd168ef5d6fd5647e832b3da4a22a34998" Jul 10 17:18:57.449477 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:57.449446 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:18:57.449592 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:57.449522 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:18:57.449592 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:57.449552 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 17:18:58.460275 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:58.460231 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" event={"ID":"c35dbe24-690a-4c79-a4da-007f9a351f09","Type":"ContainerStarted","Data":"f512e1eb6cf48cbec394707c117446bbd4e273ee8f589ad90361bb14d1421a94"} Jul 10 17:18:58.589855 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:58.589811 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:18:58.590075 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:58.590053 2585 patch_prober.go:28] interesting pod/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 10 17:18:58.590128 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:58.590112 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" podUID="c35dbe24-690a-4c79-a4da-007f9a351f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 10 17:18:59.199428 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:59.199391 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:18:59.590122 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:59.590035 2585 patch_prober.go:28] interesting pod/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 10 17:18:59.590584 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:18:59.590118 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" podUID="c35dbe24-690a-4c79-a4da-007f9a351f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 10 17:19:00.589525 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:00.589493 2585 patch_prober.go:28] interesting pod/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 10 17:19:00.589702 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:00.589546 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" podUID="c35dbe24-690a-4c79-a4da-007f9a351f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 10 17:19:01.589204 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:01.589166 2585 patch_prober.go:28] interesting pod/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 10 17:19:01.589608 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:01.589233 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" podUID="c35dbe24-690a-4c79-a4da-007f9a351f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 10 17:19:02.592388 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:02.592358 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:19:02.592869 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:02.592635 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:19:02.593395 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:02.593376 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:19:27.899349 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:27.899300 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7"] Jul 10 17:19:27.904812 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:27.904776 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:27.909213 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:27.909182 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c9789a\"/\"auth-test-model-c9789a-kserve-self-signed-certs\"" Jul 10 17:19:27.909438 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:27.909420 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c9789a\"/\"default-dockercfg-glf44\"" Jul 10 17:19:27.909546 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:27.909521 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c9789a\"/\"kube-root-ca.crt\"" Jul 10 17:19:27.909644 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:27.909629 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c9789a\"/\"openshift-service-ca.crt\"" Jul 10 17:19:27.913533 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:27.913508 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7"] Jul 10 17:19:28.032340 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.032294 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5s7v\" (UniqueName: \"kubernetes.io/projected/fa475de7-9637-4e5c-b342-980abe3cdb2e-kube-api-access-h5s7v\") pod \"auth-test-model-c9789a-kserve-74c887dc59-wp8g7\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.032608 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.032353 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-home\") pod \"auth-test-model-c9789a-kserve-74c887dc59-wp8g7\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.032608 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.032437 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-model-cache\") pod \"auth-test-model-c9789a-kserve-74c887dc59-wp8g7\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.032608 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.032492 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-tmp-dir\") pod \"auth-test-model-c9789a-kserve-74c887dc59-wp8g7\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.032608 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.032530 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa475de7-9637-4e5c-b342-980abe3cdb2e-tls-certs\") pod \"auth-test-model-c9789a-kserve-74c887dc59-wp8g7\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.032608 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.032550 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-dshm\") pod \"auth-test-model-c9789a-kserve-74c887dc59-wp8g7\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.133345 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.133307 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5s7v\" (UniqueName: \"kubernetes.io/projected/fa475de7-9637-4e5c-b342-980abe3cdb2e-kube-api-access-h5s7v\") pod \"auth-test-model-c9789a-kserve-74c887dc59-wp8g7\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.133543 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.133358 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-home\") pod \"auth-test-model-c9789a-kserve-74c887dc59-wp8g7\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.133543 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.133406 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-model-cache\") pod \"auth-test-model-c9789a-kserve-74c887dc59-wp8g7\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.133543 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.133434 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-tmp-dir\") pod \"auth-test-model-c9789a-kserve-74c887dc59-wp8g7\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.133543 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.133459 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa475de7-9637-4e5c-b342-980abe3cdb2e-tls-certs\") pod \"auth-test-model-c9789a-kserve-74c887dc59-wp8g7\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.133543 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.133483 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-dshm\") pod \"auth-test-model-c9789a-kserve-74c887dc59-wp8g7\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.133908 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.133878 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-tmp-dir\") pod \"auth-test-model-c9789a-kserve-74c887dc59-wp8g7\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.133908 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.133891 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-model-cache\") pod \"auth-test-model-c9789a-kserve-74c887dc59-wp8g7\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.134049 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.133954 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-home\") pod \"auth-test-model-c9789a-kserve-74c887dc59-wp8g7\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.135872 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.135851 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-dshm\") pod \"auth-test-model-c9789a-kserve-74c887dc59-wp8g7\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.136005 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.135990 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa475de7-9637-4e5c-b342-980abe3cdb2e-tls-certs\") pod \"auth-test-model-c9789a-kserve-74c887dc59-wp8g7\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.143190 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.143168 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5s7v\" (UniqueName: \"kubernetes.io/projected/fa475de7-9637-4e5c-b342-980abe3cdb2e-kube-api-access-h5s7v\") pod \"auth-test-model-c9789a-kserve-74c887dc59-wp8g7\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.217111 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.217010 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:28.364360 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.364333 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7"] Jul 10 17:19:28.366214 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:19:28.366184 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa475de7_9637_4e5c_b342_980abe3cdb2e.slice/crio-5ff4116f69bed5546414536a2874e7e7a7705e3f842c2c0ed6b5eb81f6e0263d WatchSource:0}: Error finding container 5ff4116f69bed5546414536a2874e7e7a7705e3f842c2c0ed6b5eb81f6e0263d: Status 404 returned error can't find the container with id 5ff4116f69bed5546414536a2874e7e7a7705e3f842c2c0ed6b5eb81f6e0263d Jul 10 17:19:28.634111 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.634078 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" event={"ID":"fa475de7-9637-4e5c-b342-980abe3cdb2e","Type":"ContainerStarted","Data":"223db3147ae3f9ad877577e0a0ea3a6a718bcc2bb6e43c82b1070ddc81bbe8db"} Jul 10 17:19:28.634287 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.634121 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" event={"ID":"fa475de7-9637-4e5c-b342-980abe3cdb2e","Type":"ContainerStarted","Data":"5ff4116f69bed5546414536a2874e7e7a7705e3f842c2c0ed6b5eb81f6e0263d"} Jul 10 17:19:28.651592 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:28.651535 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" podStartSLOduration=1.651518134 podStartE2EDuration="1.651518134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:19:28.650898604 +0000 UTC m=+2588.072775081" watchObservedRunningTime="2026-07-10 17:19:28.651518134 +0000 UTC m=+2588.073394599" Jul 10 17:19:31.993175 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:31.993142 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:19:38.217479 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:38.217443 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:38.217906 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:38.217495 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:38.229422 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:38.229399 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:38.698396 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:38.698369 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:19:57.690199 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:19:57.690164 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:20:16.594414 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:20:16.594382 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:20:33.904938 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:20:33.904903 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:20:50.098888 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:20:50.098851 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:21:04.893904 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:04.893869 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:21:20.896277 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:20.896190 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:21:21.257412 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:21.257312 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 17:21:21.269067 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:21.269041 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 17:21:35.786051 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:35.786016 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7"] Jul 10 17:21:36.994436 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:36.994397 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:21:50.797654 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:50.797611 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:21:50.807714 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:50.807679 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" podUID="fa475de7-9637-4e5c-b342-980abe3cdb2e" containerName="main" containerID="cri-o://223db3147ae3f9ad877577e0a0ea3a6a718bcc2bb6e43c82b1070ddc81bbe8db" gracePeriod=45 Jul 10 17:21:51.063607 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.063539 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:21:51.202250 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.202222 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-tmp-dir\") pod \"fa475de7-9637-4e5c-b342-980abe3cdb2e\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " Jul 10 17:21:51.202250 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.202256 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-dshm\") pod \"fa475de7-9637-4e5c-b342-980abe3cdb2e\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " Jul 10 17:21:51.202452 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.202308 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-home\") pod \"fa475de7-9637-4e5c-b342-980abe3cdb2e\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " Jul 10 17:21:51.202452 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.202370 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-model-cache\") pod \"fa475de7-9637-4e5c-b342-980abe3cdb2e\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " Jul 10 17:21:51.202452 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.202417 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5s7v\" (UniqueName: \"kubernetes.io/projected/fa475de7-9637-4e5c-b342-980abe3cdb2e-kube-api-access-h5s7v\") pod \"fa475de7-9637-4e5c-b342-980abe3cdb2e\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " Jul 10 17:21:51.202586 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.202450 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa475de7-9637-4e5c-b342-980abe3cdb2e-tls-certs\") pod \"fa475de7-9637-4e5c-b342-980abe3cdb2e\" (UID: \"fa475de7-9637-4e5c-b342-980abe3cdb2e\") " Jul 10 17:21:51.202586 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.202492 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa475de7-9637-4e5c-b342-980abe3cdb2e" (UID: "fa475de7-9637-4e5c-b342-980abe3cdb2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:51.202694 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.202582 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-home" (OuterVolumeSpecName: "home") pod "fa475de7-9637-4e5c-b342-980abe3cdb2e" (UID: "fa475de7-9637-4e5c-b342-980abe3cdb2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:51.202694 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.202625 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa475de7-9637-4e5c-b342-980abe3cdb2e" (UID: "fa475de7-9637-4e5c-b342-980abe3cdb2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:51.202898 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.202876 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-model-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:21:51.202898 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.202898 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-tmp-dir\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:21:51.203084 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.202906 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-home\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:21:51.204653 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.204628 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa475de7-9637-4e5c-b342-980abe3cdb2e-kube-api-access-h5s7v" (OuterVolumeSpecName: "kube-api-access-h5s7v") pod "fa475de7-9637-4e5c-b342-980abe3cdb2e" (UID: "fa475de7-9637-4e5c-b342-980abe3cdb2e"). InnerVolumeSpecName "kube-api-access-h5s7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:21:51.204653 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.204639 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa475de7-9637-4e5c-b342-980abe3cdb2e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa475de7-9637-4e5c-b342-980abe3cdb2e" (UID: "fa475de7-9637-4e5c-b342-980abe3cdb2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:21:51.204808 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.204632 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-dshm" (OuterVolumeSpecName: "dshm") pod "fa475de7-9637-4e5c-b342-980abe3cdb2e" (UID: "fa475de7-9637-4e5c-b342-980abe3cdb2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:21:51.304378 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.304332 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5s7v\" (UniqueName: \"kubernetes.io/projected/fa475de7-9637-4e5c-b342-980abe3cdb2e-kube-api-access-h5s7v\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:21:51.304378 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.304376 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa475de7-9637-4e5c-b342-980abe3cdb2e-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:21:51.304378 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.304389 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa475de7-9637-4e5c-b342-980abe3cdb2e-dshm\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:21:51.407759 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.407714 2585 generic.go:356] "Generic (PLEG): container finished" podID="fa475de7-9637-4e5c-b342-980abe3cdb2e" containerID="223db3147ae3f9ad877577e0a0ea3a6a718bcc2bb6e43c82b1070ddc81bbe8db" exitCode=0 Jul 10 17:21:51.407924 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.407784 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" Jul 10 17:21:51.407924 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.407801 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" event={"ID":"fa475de7-9637-4e5c-b342-980abe3cdb2e","Type":"ContainerDied","Data":"223db3147ae3f9ad877577e0a0ea3a6a718bcc2bb6e43c82b1070ddc81bbe8db"} Jul 10 17:21:51.407924 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.407849 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7" event={"ID":"fa475de7-9637-4e5c-b342-980abe3cdb2e","Type":"ContainerDied","Data":"5ff4116f69bed5546414536a2874e7e7a7705e3f842c2c0ed6b5eb81f6e0263d"} Jul 10 17:21:51.407924 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.407871 2585 scope.go:117] "RemoveContainer" containerID="223db3147ae3f9ad877577e0a0ea3a6a718bcc2bb6e43c82b1070ddc81bbe8db" Jul 10 17:21:51.416940 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.416919 2585 scope.go:117] "RemoveContainer" containerID="223db3147ae3f9ad877577e0a0ea3a6a718bcc2bb6e43c82b1070ddc81bbe8db" Jul 10 17:21:51.417202 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:21:51.417184 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"223db3147ae3f9ad877577e0a0ea3a6a718bcc2bb6e43c82b1070ddc81bbe8db\": container with ID starting with 223db3147ae3f9ad877577e0a0ea3a6a718bcc2bb6e43c82b1070ddc81bbe8db not found: ID does not exist" containerID="223db3147ae3f9ad877577e0a0ea3a6a718bcc2bb6e43c82b1070ddc81bbe8db" Jul 10 17:21:51.417255 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.417211 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"223db3147ae3f9ad877577e0a0ea3a6a718bcc2bb6e43c82b1070ddc81bbe8db"} err="failed to get container status \"223db3147ae3f9ad877577e0a0ea3a6a718bcc2bb6e43c82b1070ddc81bbe8db\": rpc error: code = NotFound desc = could not find container \"223db3147ae3f9ad877577e0a0ea3a6a718bcc2bb6e43c82b1070ddc81bbe8db\": container with ID starting with 223db3147ae3f9ad877577e0a0ea3a6a718bcc2bb6e43c82b1070ddc81bbe8db not found: ID does not exist" Jul 10 17:21:51.426084 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.426056 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7"] Jul 10 17:21:51.430090 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:51.430065 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c9789a/auth-test-model-c9789a-kserve-74c887dc59-wp8g7"] Jul 10 17:21:53.325729 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:21:53.325693 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa475de7-9637-4e5c-b342-980abe3cdb2e" path="/var/lib/kubelet/pods/fa475de7-9637-4e5c-b342-980abe3cdb2e/volumes" Jul 10 17:22:11.494775 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.494722 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl"] Jul 10 17:22:11.495260 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.495130 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa475de7-9637-4e5c-b342-980abe3cdb2e" containerName="main" Jul 10 17:22:11.495260 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.495142 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa475de7-9637-4e5c-b342-980abe3cdb2e" containerName="main" Jul 10 17:22:11.495260 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.495217 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa475de7-9637-4e5c-b342-980abe3cdb2e" containerName="main" Jul 10 17:22:11.497589 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.497568 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.498999 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.498970 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-tmp-dir\") pod \"sub-test-model-c9789a-kserve-65cc477d85-2mgrl\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.499129 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.499034 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-home\") pod \"sub-test-model-c9789a-kserve-65cc477d85-2mgrl\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.499129 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.499062 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-model-cache\") pod \"sub-test-model-c9789a-kserve-65cc477d85-2mgrl\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.499129 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.499089 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr82j\" (UniqueName: \"kubernetes.io/projected/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-kube-api-access-mr82j\") pod \"sub-test-model-c9789a-kserve-65cc477d85-2mgrl\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.499292 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.499144 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-dshm\") pod \"sub-test-model-c9789a-kserve-65cc477d85-2mgrl\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.499292 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.499168 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-tls-certs\") pod \"sub-test-model-c9789a-kserve-65cc477d85-2mgrl\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.501884 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.501864 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c9789a\"/\"kube-root-ca.crt\"" Jul 10 17:22:11.501884 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.501876 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c9789a\"/\"sub-test-model-c9789a-kserve-self-signed-certs\"" Jul 10 17:22:11.502047 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.501865 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c9789a\"/\"openshift-service-ca.crt\"" Jul 10 17:22:11.502047 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.501968 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c9789a\"/\"default-dockercfg-glf44\"" Jul 10 17:22:11.507710 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.507689 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl"] Jul 10 17:22:11.599893 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.599860 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-tmp-dir\") pod \"sub-test-model-c9789a-kserve-65cc477d85-2mgrl\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.600095 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.599905 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-home\") pod \"sub-test-model-c9789a-kserve-65cc477d85-2mgrl\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.600095 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.599923 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-model-cache\") pod \"sub-test-model-c9789a-kserve-65cc477d85-2mgrl\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.600095 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.599941 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr82j\" (UniqueName: \"kubernetes.io/projected/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-kube-api-access-mr82j\") pod \"sub-test-model-c9789a-kserve-65cc477d85-2mgrl\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.600095 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.599974 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-dshm\") pod \"sub-test-model-c9789a-kserve-65cc477d85-2mgrl\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.600095 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.599993 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-tls-certs\") pod \"sub-test-model-c9789a-kserve-65cc477d85-2mgrl\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.600392 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.600343 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-home\") pod \"sub-test-model-c9789a-kserve-65cc477d85-2mgrl\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.600439 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.600405 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-model-cache\") pod \"sub-test-model-c9789a-kserve-65cc477d85-2mgrl\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.600476 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.600397 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-tmp-dir\") pod \"sub-test-model-c9789a-kserve-65cc477d85-2mgrl\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.602181 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.602159 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-dshm\") pod \"sub-test-model-c9789a-kserve-65cc477d85-2mgrl\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.602449 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.602430 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-tls-certs\") pod \"sub-test-model-c9789a-kserve-65cc477d85-2mgrl\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.609432 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.609404 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr82j\" (UniqueName: \"kubernetes.io/projected/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-kube-api-access-mr82j\") pod \"sub-test-model-c9789a-kserve-65cc477d85-2mgrl\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.807442 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.807333 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:11.952877 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:11.952844 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl"] Jul 10 17:22:11.954684 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:22:11.954655 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd3ad52b_5a89_4e4c_969a_1b9f39d02718.slice/crio-b39c403d3b38e8b92752a0eb27017d4163770ab7ee2fac486434aadc04ca5cad WatchSource:0}: Error finding container b39c403d3b38e8b92752a0eb27017d4163770ab7ee2fac486434aadc04ca5cad: Status 404 returned error can't find the container with id b39c403d3b38e8b92752a0eb27017d4163770ab7ee2fac486434aadc04ca5cad Jul 10 17:22:12.530655 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:12.530616 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" event={"ID":"fd3ad52b-5a89-4e4c-969a-1b9f39d02718","Type":"ContainerStarted","Data":"3246566bcbee1b52a1689f022e6cbbef17918d20b4d171b26ac546d632c684c8"} Jul 10 17:22:12.531116 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:12.530662 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" event={"ID":"fd3ad52b-5a89-4e4c-969a-1b9f39d02718","Type":"ContainerStarted","Data":"b39c403d3b38e8b92752a0eb27017d4163770ab7ee2fac486434aadc04ca5cad"} Jul 10 17:22:12.553535 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:12.553473 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" podStartSLOduration=1.553456668 podStartE2EDuration="1.553456668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:22:12.550564797 +0000 UTC m=+2751.972441255" watchObservedRunningTime="2026-07-10 17:22:12.553456668 +0000 UTC m=+2751.975333152" Jul 10 17:22:16.101092 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:16.101050 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:22:21.807689 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:21.807597 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:21.808099 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:21.807917 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:21.820508 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:21.820482 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:22.599266 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:22.599230 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:22:31.002364 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:31.002331 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:22:50.495507 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:50.495467 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:22:57.299622 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:22:57.299586 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl"] Jul 10 17:23:08.793805 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:08.793770 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:23:12.319787 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.319705 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" podUID="fd3ad52b-5a89-4e4c-969a-1b9f39d02718" containerName="main" containerID="cri-o://3246566bcbee1b52a1689f022e6cbbef17918d20b4d171b26ac546d632c684c8" gracePeriod=45 Jul 10 17:23:12.575151 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.575087 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:23:12.674812 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.674773 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-dshm\") pod \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " Jul 10 17:23:12.675026 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.674837 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-tls-certs\") pod \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " Jul 10 17:23:12.675026 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.674885 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-model-cache\") pod \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " Jul 10 17:23:12.675026 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.674943 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-tmp-dir\") pod \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " Jul 10 17:23:12.675026 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.674993 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-home\") pod \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " Jul 10 17:23:12.675233 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.675044 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mr82j\" (UniqueName: \"kubernetes.io/projected/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-kube-api-access-mr82j\") pod \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\" (UID: \"fd3ad52b-5a89-4e4c-969a-1b9f39d02718\") " Jul 10 17:23:12.675283 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.675210 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd3ad52b-5a89-4e4c-969a-1b9f39d02718" (UID: "fd3ad52b-5a89-4e4c-969a-1b9f39d02718"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:23:12.675283 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.675239 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd3ad52b-5a89-4e4c-969a-1b9f39d02718" (UID: "fd3ad52b-5a89-4e4c-969a-1b9f39d02718"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:23:12.675377 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.675277 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-home" (OuterVolumeSpecName: "home") pod "fd3ad52b-5a89-4e4c-969a-1b9f39d02718" (UID: "fd3ad52b-5a89-4e4c-969a-1b9f39d02718"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:23:12.675427 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.675409 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-home\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:23:12.675464 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.675436 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-model-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:23:12.675464 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.675457 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-tmp-dir\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:23:12.676992 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.676973 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd3ad52b-5a89-4e4c-969a-1b9f39d02718" (UID: "fd3ad52b-5a89-4e4c-969a-1b9f39d02718"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:23:12.677056 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.677024 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-dshm" (OuterVolumeSpecName: "dshm") pod "fd3ad52b-5a89-4e4c-969a-1b9f39d02718" (UID: "fd3ad52b-5a89-4e4c-969a-1b9f39d02718"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:23:12.677115 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.677097 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-kube-api-access-mr82j" (OuterVolumeSpecName: "kube-api-access-mr82j") pod "fd3ad52b-5a89-4e4c-969a-1b9f39d02718" (UID: "fd3ad52b-5a89-4e4c-969a-1b9f39d02718"). InnerVolumeSpecName "kube-api-access-mr82j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:23:12.776863 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.776818 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr82j\" (UniqueName: \"kubernetes.io/projected/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-kube-api-access-mr82j\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:23:12.776863 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.776855 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-dshm\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:23:12.776863 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.776867 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd3ad52b-5a89-4e4c-969a-1b9f39d02718-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:23:12.867483 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.867450 2585 generic.go:356] "Generic (PLEG): container finished" podID="fd3ad52b-5a89-4e4c-969a-1b9f39d02718" containerID="3246566bcbee1b52a1689f022e6cbbef17918d20b4d171b26ac546d632c684c8" exitCode=0 Jul 10 17:23:12.867679 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.867489 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" Jul 10 17:23:12.867679 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.867539 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" event={"ID":"fd3ad52b-5a89-4e4c-969a-1b9f39d02718","Type":"ContainerDied","Data":"3246566bcbee1b52a1689f022e6cbbef17918d20b4d171b26ac546d632c684c8"} Jul 10 17:23:12.867679 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.867582 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl" event={"ID":"fd3ad52b-5a89-4e4c-969a-1b9f39d02718","Type":"ContainerDied","Data":"b39c403d3b38e8b92752a0eb27017d4163770ab7ee2fac486434aadc04ca5cad"} Jul 10 17:23:12.867679 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.867602 2585 scope.go:117] "RemoveContainer" containerID="3246566bcbee1b52a1689f022e6cbbef17918d20b4d171b26ac546d632c684c8" Jul 10 17:23:12.877085 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.877067 2585 scope.go:117] "RemoveContainer" containerID="3246566bcbee1b52a1689f022e6cbbef17918d20b4d171b26ac546d632c684c8" Jul 10 17:23:12.877355 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:23:12.877334 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3246566bcbee1b52a1689f022e6cbbef17918d20b4d171b26ac546d632c684c8\": container with ID starting with 3246566bcbee1b52a1689f022e6cbbef17918d20b4d171b26ac546d632c684c8 not found: ID does not exist" containerID="3246566bcbee1b52a1689f022e6cbbef17918d20b4d171b26ac546d632c684c8" Jul 10 17:23:12.877405 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.877371 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3246566bcbee1b52a1689f022e6cbbef17918d20b4d171b26ac546d632c684c8"} err="failed to get container status \"3246566bcbee1b52a1689f022e6cbbef17918d20b4d171b26ac546d632c684c8\": rpc error: code = NotFound desc = could not find container \"3246566bcbee1b52a1689f022e6cbbef17918d20b4d171b26ac546d632c684c8\": container with ID starting with 3246566bcbee1b52a1689f022e6cbbef17918d20b4d171b26ac546d632c684c8 not found: ID does not exist" Jul 10 17:23:12.889545 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.889515 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl"] Jul 10 17:23:12.893149 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:12.893126 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c9789a/sub-test-model-c9789a-kserve-65cc477d85-2mgrl"] Jul 10 17:23:13.326692 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:13.326609 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd3ad52b-5a89-4e4c-969a-1b9f39d02718" path="/var/lib/kubelet/pods/fd3ad52b-5a89-4e4c-969a-1b9f39d02718/volumes" Jul 10 17:23:17.301365 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.301331 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj"] Jul 10 17:23:17.301867 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.301837 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd3ad52b-5a89-4e4c-969a-1b9f39d02718" containerName="main" Jul 10 17:23:17.301867 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.301851 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd3ad52b-5a89-4e4c-969a-1b9f39d02718" containerName="main" Jul 10 17:23:17.301946 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.301927 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd3ad52b-5a89-4e4c-969a-1b9f39d02718" containerName="main" Jul 10 17:23:17.304543 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.304525 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.308379 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.308359 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fd7cfb\"/\"default-dockercfg-lpjn8\"" Jul 10 17:23:17.308504 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.308382 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fd7cfb\"/\"kube-root-ca.crt\"" Jul 10 17:23:17.309534 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.309511 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fd7cfb\"/\"openshift-service-ca.crt\"" Jul 10 17:23:17.309696 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.309674 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fd7cfb\"/\"rate-test-model-fd7cfb-kserve-self-signed-certs\"" Jul 10 17:23:17.317369 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.317348 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj"] Jul 10 17:23:17.418390 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.418347 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-model-cache\") pod \"rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.418390 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.418395 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7245bc9-26a7-4190-9051-15022bc0af12-tls-certs\") pod \"rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.418659 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.418452 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-tmp-dir\") pod \"rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.418659 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.418544 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-dshm\") pod \"rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.418659 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.418589 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4g6h\" (UniqueName: \"kubernetes.io/projected/c7245bc9-26a7-4190-9051-15022bc0af12-kube-api-access-n4g6h\") pod \"rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.418815 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.418685 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-home\") pod \"rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.520046 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.520001 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-dshm\") pod \"rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.520046 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.520056 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4g6h\" (UniqueName: \"kubernetes.io/projected/c7245bc9-26a7-4190-9051-15022bc0af12-kube-api-access-n4g6h\") pod \"rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.520314 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.520136 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-home\") pod \"rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.520314 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.520174 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-model-cache\") pod \"rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.520314 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.520199 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7245bc9-26a7-4190-9051-15022bc0af12-tls-certs\") pod \"rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.520314 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.520229 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-tmp-dir\") pod \"rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.520572 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.520551 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-home\") pod \"rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.520628 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.520565 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-model-cache\") pod \"rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.520628 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.520612 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-tmp-dir\") pod \"rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.522254 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.522234 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-dshm\") pod \"rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.522637 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.522616 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7245bc9-26a7-4190-9051-15022bc0af12-tls-certs\") pod \"rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.528402 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.528379 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4g6h\" (UniqueName: \"kubernetes.io/projected/c7245bc9-26a7-4190-9051-15022bc0af12-kube-api-access-n4g6h\") pod \"rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.617034 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.616999 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:17.758169 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.758138 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj"] Jul 10 17:23:17.759037 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:23:17.759009 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7245bc9_26a7_4190_9051_15022bc0af12.slice/crio-8f909d2c530b93584aeea5f09a8fe19cbb355d4d4c4471311c90c6fa50229de3 WatchSource:0}: Error finding container 8f909d2c530b93584aeea5f09a8fe19cbb355d4d4c4471311c90c6fa50229de3: Status 404 returned error can't find the container with id 8f909d2c530b93584aeea5f09a8fe19cbb355d4d4c4471311c90c6fa50229de3 Jul 10 17:23:17.899330 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.899239 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" event={"ID":"c7245bc9-26a7-4190-9051-15022bc0af12","Type":"ContainerStarted","Data":"c542741cb9523fddb536cf9b378a7e27f6cd033af0fb09b0eec0332c0d1f4a8d"} Jul 10 17:23:17.899330 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.899284 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" event={"ID":"c7245bc9-26a7-4190-9051-15022bc0af12","Type":"ContainerStarted","Data":"8f909d2c530b93584aeea5f09a8fe19cbb355d4d4c4471311c90c6fa50229de3"} Jul 10 17:23:17.919332 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:17.919272 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" podStartSLOduration=0.919255566 podStartE2EDuration="919.255566ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:23:17.916182918 +0000 UTC m=+2817.338059386" watchObservedRunningTime="2026-07-10 17:23:17.919255566 +0000 UTC m=+2817.341132031" Jul 10 17:23:21.389469 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:21.389429 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:23:27.617457 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:27.617414 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:27.617457 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:27.617459 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:27.630284 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:27.630257 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:23:27.965817 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:23:27.965706 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:24:09.758572 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:09.758536 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:24:24.393481 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:24.393397 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:24:44.697339 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:44.697298 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:24:48.940947 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:48.940915 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj"] Jul 10 17:24:54.065689 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:54.065646 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7504-139898138368944"] Jul 10 17:24:54.069382 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:54.069359 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7504-139898138368944" Jul 10 17:24:54.072182 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:54.072154 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-dkfct\"" Jul 10 17:24:54.076214 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:54.076191 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7504-139898138368944"] Jul 10 17:24:54.102726 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:54.102705 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krt6h\" (UniqueName: \"kubernetes.io/projected/6c26499d-f083-4a20-8ae5-f6f518c779a2-kube-api-access-krt6h\") pod \"test-curl-7504-139898138368944\" (UID: \"6c26499d-f083-4a20-8ae5-f6f518c779a2\") " pod="openshift-ingress/test-curl-7504-139898138368944" Jul 10 17:24:54.204142 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:54.204118 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krt6h\" (UniqueName: \"kubernetes.io/projected/6c26499d-f083-4a20-8ae5-f6f518c779a2-kube-api-access-krt6h\") pod \"test-curl-7504-139898138368944\" (UID: \"6c26499d-f083-4a20-8ae5-f6f518c779a2\") " pod="openshift-ingress/test-curl-7504-139898138368944" Jul 10 17:24:54.212632 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:54.212611 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krt6h\" (UniqueName: \"kubernetes.io/projected/6c26499d-f083-4a20-8ae5-f6f518c779a2-kube-api-access-krt6h\") pod \"test-curl-7504-139898138368944\" (UID: \"6c26499d-f083-4a20-8ae5-f6f518c779a2\") " pod="openshift-ingress/test-curl-7504-139898138368944" Jul 10 17:24:54.380326 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:54.380309 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7504-139898138368944" Jul 10 17:24:54.510335 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:54.510311 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7504-139898138368944"] Jul 10 17:24:54.512230 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:24:54.512205 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c26499d_f083_4a20_8ae5_f6f518c779a2.slice/crio-a86f0d8e7d2adee07562cdbc514b452c77d98d07fa9f92116fe31e917a360d53 WatchSource:0}: Error finding container a86f0d8e7d2adee07562cdbc514b452c77d98d07fa9f92116fe31e917a360d53: Status 404 returned error can't find the container with id a86f0d8e7d2adee07562cdbc514b452c77d98d07fa9f92116fe31e917a360d53 Jul 10 17:24:54.513480 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:54.513459 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:24:55.428702 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:55.428670 2585 generic.go:356] "Generic (PLEG): container finished" podID="6c26499d-f083-4a20-8ae5-f6f518c779a2" containerID="de12c303704874de42dfabcf52bd8a159a5d786ad9e4e85800b47af5dc58d848" exitCode=0 Jul 10 17:24:55.429114 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:55.428777 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7504-139898138368944" event={"ID":"6c26499d-f083-4a20-8ae5-f6f518c779a2","Type":"ContainerDied","Data":"de12c303704874de42dfabcf52bd8a159a5d786ad9e4e85800b47af5dc58d848"} Jul 10 17:24:55.429114 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:55.428828 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7504-139898138368944" event={"ID":"6c26499d-f083-4a20-8ae5-f6f518c779a2","Type":"ContainerStarted","Data":"a86f0d8e7d2adee07562cdbc514b452c77d98d07fa9f92116fe31e917a360d53"} Jul 10 17:24:56.577198 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:56.577171 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7504-139898138368944" Jul 10 17:24:56.616703 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:56.616671 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7504-139898138368944_6c26499d-f083-4a20-8ae5-f6f518c779a2/test-curl-7504-139898138368944/0.log" Jul 10 17:24:56.626967 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:56.626948 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krt6h\" (UniqueName: \"kubernetes.io/projected/6c26499d-f083-4a20-8ae5-f6f518c779a2-kube-api-access-krt6h\") pod \"6c26499d-f083-4a20-8ae5-f6f518c779a2\" (UID: \"6c26499d-f083-4a20-8ae5-f6f518c779a2\") " Jul 10 17:24:56.628809 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:56.628787 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c26499d-f083-4a20-8ae5-f6f518c779a2-kube-api-access-krt6h" (OuterVolumeSpecName: "kube-api-access-krt6h") pod "6c26499d-f083-4a20-8ae5-f6f518c779a2" (UID: "6c26499d-f083-4a20-8ae5-f6f518c779a2"). InnerVolumeSpecName "kube-api-access-krt6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:24:56.641264 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:56.641242 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7504-139898138368944"] Jul 10 17:24:56.645507 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:56.645488 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7504-139898138368944"] Jul 10 17:24:56.728148 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:56.728072 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krt6h\" (UniqueName: \"kubernetes.io/projected/6c26499d-f083-4a20-8ae5-f6f518c779a2-kube-api-access-krt6h\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:24:56.799320 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:56.799285 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7504-139898144299888"] Jul 10 17:24:56.799730 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:56.799714 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c26499d-f083-4a20-8ae5-f6f518c779a2" containerName="test-curl-7504-139898138368944" Jul 10 17:24:56.799730 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:56.799730 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c26499d-f083-4a20-8ae5-f6f518c779a2" containerName="test-curl-7504-139898138368944" Jul 10 17:24:56.799891 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:56.799824 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c26499d-f083-4a20-8ae5-f6f518c779a2" containerName="test-curl-7504-139898138368944" Jul 10 17:24:56.802916 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:56.802899 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7504-139898144299888" Jul 10 17:24:56.806454 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:56.806427 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7504-139898144299888"] Jul 10 17:24:56.929819 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:56.929794 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22c87\" (UniqueName: \"kubernetes.io/projected/126fa203-2114-4ead-90f6-4ee85514cc08-kube-api-access-22c87\") pod \"test-curl-7504-139898144299888\" (UID: \"126fa203-2114-4ead-90f6-4ee85514cc08\") " pod="openshift-ingress/test-curl-7504-139898144299888" Jul 10 17:24:57.031111 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:57.031053 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22c87\" (UniqueName: \"kubernetes.io/projected/126fa203-2114-4ead-90f6-4ee85514cc08-kube-api-access-22c87\") pod \"test-curl-7504-139898144299888\" (UID: \"126fa203-2114-4ead-90f6-4ee85514cc08\") " pod="openshift-ingress/test-curl-7504-139898144299888" Jul 10 17:24:57.039129 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:57.039107 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22c87\" (UniqueName: \"kubernetes.io/projected/126fa203-2114-4ead-90f6-4ee85514cc08-kube-api-access-22c87\") pod \"test-curl-7504-139898144299888\" (UID: \"126fa203-2114-4ead-90f6-4ee85514cc08\") " pod="openshift-ingress/test-curl-7504-139898144299888" Jul 10 17:24:57.113050 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:57.113026 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7504-139898144299888" Jul 10 17:24:57.241700 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:57.241673 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7504-139898144299888"] Jul 10 17:24:57.242598 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:24:57.242574 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod126fa203_2114_4ead_90f6_4ee85514cc08.slice/crio-cfd243e3544fcce927ab804a55063c0bf924ddb9f25f2550edcb012e6c8c5fd1 WatchSource:0}: Error finding container cfd243e3544fcce927ab804a55063c0bf924ddb9f25f2550edcb012e6c8c5fd1: Status 404 returned error can't find the container with id cfd243e3544fcce927ab804a55063c0bf924ddb9f25f2550edcb012e6c8c5fd1 Jul 10 17:24:57.325494 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:57.325437 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c26499d-f083-4a20-8ae5-f6f518c779a2" path="/var/lib/kubelet/pods/6c26499d-f083-4a20-8ae5-f6f518c779a2/volumes" Jul 10 17:24:57.440514 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:57.440497 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7504-139898138368944" Jul 10 17:24:57.440600 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:57.440543 2585 scope.go:117] "RemoveContainer" containerID="de12c303704874de42dfabcf52bd8a159a5d786ad9e4e85800b47af5dc58d848" Jul 10 17:24:57.441787 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:57.441758 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7504-139898144299888" event={"ID":"126fa203-2114-4ead-90f6-4ee85514cc08","Type":"ContainerStarted","Data":"cfd243e3544fcce927ab804a55063c0bf924ddb9f25f2550edcb012e6c8c5fd1"} Jul 10 17:24:58.449630 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:58.449597 2585 generic.go:356] "Generic (PLEG): container finished" podID="126fa203-2114-4ead-90f6-4ee85514cc08" containerID="97b57418ded2ca3eee7cb8845c36cdc8358bba971508779a0e50c9621350417d" exitCode=0 Jul 10 17:24:58.449994 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:58.449647 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7504-139898144299888" event={"ID":"126fa203-2114-4ead-90f6-4ee85514cc08","Type":"ContainerDied","Data":"97b57418ded2ca3eee7cb8845c36cdc8358bba971508779a0e50c9621350417d"} Jul 10 17:24:59.585110 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:59.585088 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7504-139898144299888" Jul 10 17:24:59.605311 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:59.605292 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7504-139898144299888_126fa203-2114-4ead-90f6-4ee85514cc08/test-curl-7504-139898144299888/0.log" Jul 10 17:24:59.627862 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:59.627841 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7504-139898144299888"] Jul 10 17:24:59.630051 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:59.630031 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7504-139898144299888"] Jul 10 17:24:59.652546 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:59.652513 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22c87\" (UniqueName: \"kubernetes.io/projected/126fa203-2114-4ead-90f6-4ee85514cc08-kube-api-access-22c87\") pod \"126fa203-2114-4ead-90f6-4ee85514cc08\" (UID: \"126fa203-2114-4ead-90f6-4ee85514cc08\") " Jul 10 17:24:59.654402 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:59.654381 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126fa203-2114-4ead-90f6-4ee85514cc08-kube-api-access-22c87" (OuterVolumeSpecName: "kube-api-access-22c87") pod "126fa203-2114-4ead-90f6-4ee85514cc08" (UID: "126fa203-2114-4ead-90f6-4ee85514cc08"). InnerVolumeSpecName "kube-api-access-22c87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:24:59.753177 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:59.753126 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22c87\" (UniqueName: \"kubernetes.io/projected/126fa203-2114-4ead-90f6-4ee85514cc08-kube-api-access-22c87\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:24:59.795189 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:24:59.795162 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:25:00.460538 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:00.460513 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7504-139898144299888" Jul 10 17:25:00.460713 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:00.460541 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfd243e3544fcce927ab804a55063c0bf924ddb9f25f2550edcb012e6c8c5fd1" Jul 10 17:25:00.470448 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:25:00.470422 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7504-139898144299888\" is forbidden: User \"system:node:ip-10-0-134-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-41.ec2.internal' and this object" podUID="126fa203-2114-4ead-90f6-4ee85514cc08" pod="openshift-ingress/test-curl-7504-139898144299888" Jul 10 17:25:01.330288 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:25:01.330251 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7504-139898144299888\" is forbidden: User \"system:node:ip-10-0-134-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-41.ec2.internal' and this object" podUID="126fa203-2114-4ead-90f6-4ee85514cc08" pod="openshift-ingress/test-curl-7504-139898144299888" Jul 10 17:25:01.332405 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:01.332382 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="126fa203-2114-4ead-90f6-4ee85514cc08" path="/var/lib/kubelet/pods/126fa203-2114-4ead-90f6-4ee85514cc08/volumes" Jul 10 17:25:02.130269 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:02.130238 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7504-139898144300896"] Jul 10 17:25:02.130644 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:02.130631 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="126fa203-2114-4ead-90f6-4ee85514cc08" containerName="test-curl-7504-139898144299888" Jul 10 17:25:02.130682 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:02.130647 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="126fa203-2114-4ead-90f6-4ee85514cc08" containerName="test-curl-7504-139898144299888" Jul 10 17:25:02.130717 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:02.130708 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="126fa203-2114-4ead-90f6-4ee85514cc08" containerName="test-curl-7504-139898144299888" Jul 10 17:25:02.133888 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:02.133871 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7504-139898144300896" Jul 10 17:25:02.136671 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:02.136651 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-dkfct\"" Jul 10 17:25:02.138572 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:02.138545 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7504-139898144300896"] Jul 10 17:25:02.175962 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:02.175937 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x79tj\" (UniqueName: \"kubernetes.io/projected/213c8598-95f0-42a7-8823-0c3c543fcd65-kube-api-access-x79tj\") pod \"test-curl-7504-139898144300896\" (UID: \"213c8598-95f0-42a7-8823-0c3c543fcd65\") " pod="openshift-ingress/test-curl-7504-139898144300896" Jul 10 17:25:02.276707 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:02.276687 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x79tj\" (UniqueName: \"kubernetes.io/projected/213c8598-95f0-42a7-8823-0c3c543fcd65-kube-api-access-x79tj\") pod \"test-curl-7504-139898144300896\" (UID: \"213c8598-95f0-42a7-8823-0c3c543fcd65\") " pod="openshift-ingress/test-curl-7504-139898144300896" Jul 10 17:25:02.285327 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:02.285309 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x79tj\" (UniqueName: \"kubernetes.io/projected/213c8598-95f0-42a7-8823-0c3c543fcd65-kube-api-access-x79tj\") pod \"test-curl-7504-139898144300896\" (UID: \"213c8598-95f0-42a7-8823-0c3c543fcd65\") " pod="openshift-ingress/test-curl-7504-139898144300896" Jul 10 17:25:02.444554 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:02.444501 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7504-139898144300896" Jul 10 17:25:02.574236 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:02.574209 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7504-139898144300896"] Jul 10 17:25:02.574989 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:25:02.574960 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod213c8598_95f0_42a7_8823_0c3c543fcd65.slice/crio-49bc47eeac54a3b06e2f03308df103114c0c5367bfd6a61d47a54f874b533a5c WatchSource:0}: Error finding container 49bc47eeac54a3b06e2f03308df103114c0c5367bfd6a61d47a54f874b533a5c: Status 404 returned error can't find the container with id 49bc47eeac54a3b06e2f03308df103114c0c5367bfd6a61d47a54f874b533a5c Jul 10 17:25:03.480217 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:03.480184 2585 generic.go:356] "Generic (PLEG): container finished" podID="213c8598-95f0-42a7-8823-0c3c543fcd65" containerID="ae13a9ab87e7495841365c0abc3fadb6b34b377cf51978ad4510cdbb4c0192af" exitCode=0 Jul 10 17:25:03.480555 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:03.480257 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7504-139898144300896" event={"ID":"213c8598-95f0-42a7-8823-0c3c543fcd65","Type":"ContainerDied","Data":"ae13a9ab87e7495841365c0abc3fadb6b34b377cf51978ad4510cdbb4c0192af"} Jul 10 17:25:03.480555 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:03.480289 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7504-139898144300896" event={"ID":"213c8598-95f0-42a7-8823-0c3c543fcd65","Type":"ContainerStarted","Data":"49bc47eeac54a3b06e2f03308df103114c0c5367bfd6a61d47a54f874b533a5c"} Jul 10 17:25:03.960997 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:03.960952 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" podUID="c7245bc9-26a7-4190-9051-15022bc0af12" containerName="main" containerID="cri-o://c542741cb9523fddb536cf9b378a7e27f6cd033af0fb09b0eec0332c0d1f4a8d" gracePeriod=45 Jul 10 17:25:04.220853 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.220799 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:25:04.292274 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.292249 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-tmp-dir\") pod \"c7245bc9-26a7-4190-9051-15022bc0af12\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " Jul 10 17:25:04.292412 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.292296 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-model-cache\") pod \"c7245bc9-26a7-4190-9051-15022bc0af12\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " Jul 10 17:25:04.292412 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.292326 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-home\") pod \"c7245bc9-26a7-4190-9051-15022bc0af12\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " Jul 10 17:25:04.292412 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.292363 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4g6h\" (UniqueName: \"kubernetes.io/projected/c7245bc9-26a7-4190-9051-15022bc0af12-kube-api-access-n4g6h\") pod \"c7245bc9-26a7-4190-9051-15022bc0af12\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " Jul 10 17:25:04.292579 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.292414 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-dshm\") pod \"c7245bc9-26a7-4190-9051-15022bc0af12\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " Jul 10 17:25:04.292579 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.292463 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7245bc9-26a7-4190-9051-15022bc0af12-tls-certs\") pod \"c7245bc9-26a7-4190-9051-15022bc0af12\" (UID: \"c7245bc9-26a7-4190-9051-15022bc0af12\") " Jul 10 17:25:04.292579 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.292529 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7245bc9-26a7-4190-9051-15022bc0af12" (UID: "c7245bc9-26a7-4190-9051-15022bc0af12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:25:04.292579 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.292560 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-home" (OuterVolumeSpecName: "home") pod "c7245bc9-26a7-4190-9051-15022bc0af12" (UID: "c7245bc9-26a7-4190-9051-15022bc0af12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:25:04.292783 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.292617 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7245bc9-26a7-4190-9051-15022bc0af12" (UID: "c7245bc9-26a7-4190-9051-15022bc0af12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:25:04.292851 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.292830 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-model-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:25:04.292925 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.292854 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-home\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:25:04.292925 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.292863 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-tmp-dir\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:25:04.294552 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.294531 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-dshm" (OuterVolumeSpecName: "dshm") pod "c7245bc9-26a7-4190-9051-15022bc0af12" (UID: "c7245bc9-26a7-4190-9051-15022bc0af12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:25:04.294666 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.294548 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7245bc9-26a7-4190-9051-15022bc0af12-kube-api-access-n4g6h" (OuterVolumeSpecName: "kube-api-access-n4g6h") pod "c7245bc9-26a7-4190-9051-15022bc0af12" (UID: "c7245bc9-26a7-4190-9051-15022bc0af12"). InnerVolumeSpecName "kube-api-access-n4g6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:25:04.294666 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.294566 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7245bc9-26a7-4190-9051-15022bc0af12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7245bc9-26a7-4190-9051-15022bc0af12" (UID: "c7245bc9-26a7-4190-9051-15022bc0af12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:25:04.393232 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.393209 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7245bc9-26a7-4190-9051-15022bc0af12-dshm\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:25:04.393232 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.393229 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7245bc9-26a7-4190-9051-15022bc0af12-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:25:04.393353 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.393239 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4g6h\" (UniqueName: \"kubernetes.io/projected/c7245bc9-26a7-4190-9051-15022bc0af12-kube-api-access-n4g6h\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:25:04.487137 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.487076 2585 generic.go:356] "Generic (PLEG): container finished" podID="c7245bc9-26a7-4190-9051-15022bc0af12" containerID="c542741cb9523fddb536cf9b378a7e27f6cd033af0fb09b0eec0332c0d1f4a8d" exitCode=0 Jul 10 17:25:04.487137 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.487115 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" Jul 10 17:25:04.487535 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.487170 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" event={"ID":"c7245bc9-26a7-4190-9051-15022bc0af12","Type":"ContainerDied","Data":"c542741cb9523fddb536cf9b378a7e27f6cd033af0fb09b0eec0332c0d1f4a8d"} Jul 10 17:25:04.487535 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.487205 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj" event={"ID":"c7245bc9-26a7-4190-9051-15022bc0af12","Type":"ContainerDied","Data":"8f909d2c530b93584aeea5f09a8fe19cbb355d4d4c4471311c90c6fa50229de3"} Jul 10 17:25:04.487535 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.487221 2585 scope.go:117] "RemoveContainer" containerID="c542741cb9523fddb536cf9b378a7e27f6cd033af0fb09b0eec0332c0d1f4a8d" Jul 10 17:25:04.498096 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.498079 2585 scope.go:117] "RemoveContainer" containerID="c542741cb9523fddb536cf9b378a7e27f6cd033af0fb09b0eec0332c0d1f4a8d" Jul 10 17:25:04.498326 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:25:04.498310 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c542741cb9523fddb536cf9b378a7e27f6cd033af0fb09b0eec0332c0d1f4a8d\": container with ID starting with c542741cb9523fddb536cf9b378a7e27f6cd033af0fb09b0eec0332c0d1f4a8d not found: ID does not exist" containerID="c542741cb9523fddb536cf9b378a7e27f6cd033af0fb09b0eec0332c0d1f4a8d" Jul 10 17:25:04.498388 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.498332 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c542741cb9523fddb536cf9b378a7e27f6cd033af0fb09b0eec0332c0d1f4a8d"} err="failed to get container status \"c542741cb9523fddb536cf9b378a7e27f6cd033af0fb09b0eec0332c0d1f4a8d\": rpc error: code = NotFound desc = could not find container \"c542741cb9523fddb536cf9b378a7e27f6cd033af0fb09b0eec0332c0d1f4a8d\": container with ID starting with c542741cb9523fddb536cf9b378a7e27f6cd033af0fb09b0eec0332c0d1f4a8d not found: ID does not exist" Jul 10 17:25:04.512174 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.512151 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj"] Jul 10 17:25:04.515950 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.515926 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-fd7cfb/rate-test-model-fd7cfb-kserve-84c48b9fff-vvxxj"] Jul 10 17:25:04.637997 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.637971 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7504-139898144300896" Jul 10 17:25:04.658353 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.658327 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7504-139898144300896_213c8598-95f0-42a7-8823-0c3c543fcd65/test-curl-7504-139898144300896/0.log" Jul 10 17:25:04.684948 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.684918 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7504-139898144300896"] Jul 10 17:25:04.690936 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.690914 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7504-139898144300896"] Jul 10 17:25:04.696002 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.695979 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x79tj\" (UniqueName: \"kubernetes.io/projected/213c8598-95f0-42a7-8823-0c3c543fcd65-kube-api-access-x79tj\") pod \"213c8598-95f0-42a7-8823-0c3c543fcd65\" (UID: \"213c8598-95f0-42a7-8823-0c3c543fcd65\") " Jul 10 17:25:04.697959 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.697934 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213c8598-95f0-42a7-8823-0c3c543fcd65-kube-api-access-x79tj" (OuterVolumeSpecName: "kube-api-access-x79tj") pod "213c8598-95f0-42a7-8823-0c3c543fcd65" (UID: "213c8598-95f0-42a7-8823-0c3c543fcd65"). InnerVolumeSpecName "kube-api-access-x79tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:25:04.797006 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.796932 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x79tj\" (UniqueName: \"kubernetes.io/projected/213c8598-95f0-42a7-8823-0c3c543fcd65-kube-api-access-x79tj\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:25:04.809666 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.809632 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7504-139898144300896"] Jul 10 17:25:04.810046 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.810030 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7245bc9-26a7-4190-9051-15022bc0af12" containerName="main" Jul 10 17:25:04.810105 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.810048 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7245bc9-26a7-4190-9051-15022bc0af12" containerName="main" Jul 10 17:25:04.810105 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.810064 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="213c8598-95f0-42a7-8823-0c3c543fcd65" containerName="test-curl-7504-139898144300896" Jul 10 17:25:04.810105 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.810071 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="213c8598-95f0-42a7-8823-0c3c543fcd65" containerName="test-curl-7504-139898144300896" Jul 10 17:25:04.810203 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.810141 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7245bc9-26a7-4190-9051-15022bc0af12" containerName="main" Jul 10 17:25:04.810203 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.810149 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="213c8598-95f0-42a7-8823-0c3c543fcd65" containerName="test-curl-7504-139898144300896" Jul 10 17:25:04.813022 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.813006 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7504-139898144300896" Jul 10 17:25:04.820798 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.820776 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7504-139898144300896"] Jul 10 17:25:04.898321 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.898294 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4qgs\" (UniqueName: \"kubernetes.io/projected/3aa694c7-5602-432a-8a14-64e32f09197b-kube-api-access-s4qgs\") pod \"test-curl-7504-139898144300896\" (UID: \"3aa694c7-5602-432a-8a14-64e32f09197b\") " pod="openshift-ingress/test-curl-7504-139898144300896" Jul 10 17:25:04.999715 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:04.999689 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4qgs\" (UniqueName: \"kubernetes.io/projected/3aa694c7-5602-432a-8a14-64e32f09197b-kube-api-access-s4qgs\") pod \"test-curl-7504-139898144300896\" (UID: \"3aa694c7-5602-432a-8a14-64e32f09197b\") " pod="openshift-ingress/test-curl-7504-139898144300896" Jul 10 17:25:05.009071 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:05.009042 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4qgs\" (UniqueName: \"kubernetes.io/projected/3aa694c7-5602-432a-8a14-64e32f09197b-kube-api-access-s4qgs\") pod \"test-curl-7504-139898144300896\" (UID: \"3aa694c7-5602-432a-8a14-64e32f09197b\") " pod="openshift-ingress/test-curl-7504-139898144300896" Jul 10 17:25:05.122990 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:05.122961 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7504-139898144300896" Jul 10 17:25:05.256048 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:05.256019 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7504-139898144300896"] Jul 10 17:25:05.257057 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:25:05.257029 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3aa694c7_5602_432a_8a14_64e32f09197b.slice/crio-ce079c902964303a0c116f4ec6e70527de90a04ee1249a5972dfe474b7d9a2c3 WatchSource:0}: Error finding container ce079c902964303a0c116f4ec6e70527de90a04ee1249a5972dfe474b7d9a2c3: Status 404 returned error can't find the container with id ce079c902964303a0c116f4ec6e70527de90a04ee1249a5972dfe474b7d9a2c3 Jul 10 17:25:05.325732 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:05.325697 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="213c8598-95f0-42a7-8823-0c3c543fcd65" path="/var/lib/kubelet/pods/213c8598-95f0-42a7-8823-0c3c543fcd65/volumes" Jul 10 17:25:05.326135 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:05.326120 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7245bc9-26a7-4190-9051-15022bc0af12" path="/var/lib/kubelet/pods/c7245bc9-26a7-4190-9051-15022bc0af12/volumes" Jul 10 17:25:05.494729 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:05.494642 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7504-139898144300896" event={"ID":"3aa694c7-5602-432a-8a14-64e32f09197b","Type":"ContainerStarted","Data":"ce079c902964303a0c116f4ec6e70527de90a04ee1249a5972dfe474b7d9a2c3"} Jul 10 17:25:05.495719 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:05.495703 2585 scope.go:117] "RemoveContainer" containerID="ae13a9ab87e7495841365c0abc3fadb6b34b377cf51978ad4510cdbb4c0192af" Jul 10 17:25:05.495824 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:05.495734 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7504-139898144300896" Jul 10 17:25:06.503797 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:06.503765 2585 generic.go:356] "Generic (PLEG): container finished" podID="3aa694c7-5602-432a-8a14-64e32f09197b" containerID="4f6c32f091afe7b91071a97fe3ac82e083337d279f2770cced446272bc5014bd" exitCode=0 Jul 10 17:25:06.504198 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:06.503814 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7504-139898144300896" event={"ID":"3aa694c7-5602-432a-8a14-64e32f09197b","Type":"ContainerDied","Data":"4f6c32f091afe7b91071a97fe3ac82e083337d279f2770cced446272bc5014bd"} Jul 10 17:25:07.642681 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:07.642655 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7504-139898144300896" Jul 10 17:25:07.665655 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:07.665630 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7504-139898144300896_3aa694c7-5602-432a-8a14-64e32f09197b/test-curl-7504-139898144300896/0.log" Jul 10 17:25:07.690116 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:07.690083 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7504-139898144300896"] Jul 10 17:25:07.693599 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:07.693576 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7504-139898144300896"] Jul 10 17:25:07.724489 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:07.724459 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4qgs\" (UniqueName: \"kubernetes.io/projected/3aa694c7-5602-432a-8a14-64e32f09197b-kube-api-access-s4qgs\") pod \"3aa694c7-5602-432a-8a14-64e32f09197b\" (UID: \"3aa694c7-5602-432a-8a14-64e32f09197b\") " Jul 10 17:25:07.726648 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:07.726627 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa694c7-5602-432a-8a14-64e32f09197b-kube-api-access-s4qgs" (OuterVolumeSpecName: "kube-api-access-s4qgs") pod "3aa694c7-5602-432a-8a14-64e32f09197b" (UID: "3aa694c7-5602-432a-8a14-64e32f09197b"). InnerVolumeSpecName "kube-api-access-s4qgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:25:07.825412 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:07.825320 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4qgs\" (UniqueName: \"kubernetes.io/projected/3aa694c7-5602-432a-8a14-64e32f09197b-kube-api-access-s4qgs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:25:08.515690 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:08.515658 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7504-139898144300896" Jul 10 17:25:08.515690 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:08.515673 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce079c902964303a0c116f4ec6e70527de90a04ee1249a5972dfe474b7d9a2c3" Jul 10 17:25:08.526261 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:25:08.526230 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7504-139898144300896\" is forbidden: User \"system:node:ip-10-0-134-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-41.ec2.internal' and this object" podUID="3aa694c7-5602-432a-8a14-64e32f09197b" pod="openshift-ingress/test-curl-7504-139898144300896" Jul 10 17:25:09.326006 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:09.325970 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aa694c7-5602-432a-8a14-64e32f09197b" path="/var/lib/kubelet/pods/3aa694c7-5602-432a-8a14-64e32f09197b/volumes" Jul 10 17:25:21.989890 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:21.989855 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:25:33.633109 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:33.633074 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr"] Jul 10 17:25:33.633527 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:33.633279 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" podUID="07e2b632-14ce-43c9-89d8-7daae3e77154" containerName="istio-proxy" containerID="cri-o://16d1416b69a167fe8106ae1432503c5ffd5684d6dffb1bb511142a2205f74aeb" gracePeriod=30 Jul 10 17:25:36.130639 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:36.130603 2585 patch_prober.go:28] interesting pod/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 17:25:36.131132 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:36.130680 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" podUID="07e2b632-14ce-43c9-89d8-7daae3e77154" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 17:25:38.891200 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:38.891171 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:25:39.028868 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.028786 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-envoy\") pod \"07e2b632-14ce-43c9-89d8-7daae3e77154\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " Jul 10 17:25:39.028868 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.028854 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-workload-socket\") pod \"07e2b632-14ce-43c9-89d8-7daae3e77154\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " Jul 10 17:25:39.029096 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.028881 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e2b632-14ce-43c9-89d8-7daae3e77154-istiod-ca-cert\") pod \"07e2b632-14ce-43c9-89d8-7daae3e77154\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " Jul 10 17:25:39.029096 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.028935 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-token\") pod \"07e2b632-14ce-43c9-89d8-7daae3e77154\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " Jul 10 17:25:39.029096 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.028961 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-credential-socket\") pod \"07e2b632-14ce-43c9-89d8-7daae3e77154\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " Jul 10 17:25:39.029096 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.028993 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-workload-certs\") pod \"07e2b632-14ce-43c9-89d8-7daae3e77154\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " Jul 10 17:25:39.029096 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.029042 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb6xb\" (UniqueName: \"kubernetes.io/projected/07e2b632-14ce-43c9-89d8-7daae3e77154-kube-api-access-wb6xb\") pod \"07e2b632-14ce-43c9-89d8-7daae3e77154\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " Jul 10 17:25:39.029350 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.029134 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-data\") pod \"07e2b632-14ce-43c9-89d8-7daae3e77154\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " Jul 10 17:25:39.029350 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.029151 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "07e2b632-14ce-43c9-89d8-7daae3e77154" (UID: "07e2b632-14ce-43c9-89d8-7daae3e77154"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:25:39.029350 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.029168 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-podinfo\") pod \"07e2b632-14ce-43c9-89d8-7daae3e77154\" (UID: \"07e2b632-14ce-43c9-89d8-7daae3e77154\") " Jul 10 17:25:39.029350 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.029274 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "07e2b632-14ce-43c9-89d8-7daae3e77154" (UID: "07e2b632-14ce-43c9-89d8-7daae3e77154"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:25:39.029562 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.029382 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e2b632-14ce-43c9-89d8-7daae3e77154-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "07e2b632-14ce-43c9-89d8-7daae3e77154" (UID: "07e2b632-14ce-43c9-89d8-7daae3e77154"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:25:39.029618 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.029560 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-workload-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:25:39.029618 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.029581 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e2b632-14ce-43c9-89d8-7daae3e77154-istiod-ca-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:25:39.029618 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.029594 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-credential-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:25:39.029876 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.029851 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "07e2b632-14ce-43c9-89d8-7daae3e77154" (UID: "07e2b632-14ce-43c9-89d8-7daae3e77154"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:25:39.030061 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.030034 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-data" (OuterVolumeSpecName: "istio-data") pod "07e2b632-14ce-43c9-89d8-7daae3e77154" (UID: "07e2b632-14ce-43c9-89d8-7daae3e77154"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:25:39.031238 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.031212 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-token" (OuterVolumeSpecName: "istio-token") pod "07e2b632-14ce-43c9-89d8-7daae3e77154" (UID: "07e2b632-14ce-43c9-89d8-7daae3e77154"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:25:39.031339 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.031237 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e2b632-14ce-43c9-89d8-7daae3e77154-kube-api-access-wb6xb" (OuterVolumeSpecName: "kube-api-access-wb6xb") pod "07e2b632-14ce-43c9-89d8-7daae3e77154" (UID: "07e2b632-14ce-43c9-89d8-7daae3e77154"). InnerVolumeSpecName "kube-api-access-wb6xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:25:39.031450 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.031431 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "07e2b632-14ce-43c9-89d8-7daae3e77154" (UID: "07e2b632-14ce-43c9-89d8-7daae3e77154"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:25:39.031450 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.031440 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "07e2b632-14ce-43c9-89d8-7daae3e77154" (UID: "07e2b632-14ce-43c9-89d8-7daae3e77154"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:25:39.131276 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.131236 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-envoy\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:25:39.131276 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.131274 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-token\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:25:39.131472 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.131287 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-workload-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:25:39.131472 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.131299 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb6xb\" (UniqueName: \"kubernetes.io/projected/07e2b632-14ce-43c9-89d8-7daae3e77154-kube-api-access-wb6xb\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:25:39.131472 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.131314 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-data\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:25:39.131472 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.131326 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e2b632-14ce-43c9-89d8-7daae3e77154-istio-podinfo\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:25:39.700842 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.700806 2585 generic.go:356] "Generic (PLEG): container finished" podID="07e2b632-14ce-43c9-89d8-7daae3e77154" containerID="16d1416b69a167fe8106ae1432503c5ffd5684d6dffb1bb511142a2205f74aeb" exitCode=0 Jul 10 17:25:39.701019 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.700854 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" Jul 10 17:25:39.701019 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.700865 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" event={"ID":"07e2b632-14ce-43c9-89d8-7daae3e77154","Type":"ContainerDied","Data":"16d1416b69a167fe8106ae1432503c5ffd5684d6dffb1bb511142a2205f74aeb"} Jul 10 17:25:39.701019 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.700912 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr" event={"ID":"07e2b632-14ce-43c9-89d8-7daae3e77154","Type":"ContainerDied","Data":"5c7fd472850be0c3fd3289db4598295ea7224c417566ae5bc096c10a879e85ba"} Jul 10 17:25:39.701019 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.700933 2585 scope.go:117] "RemoveContainer" containerID="16d1416b69a167fe8106ae1432503c5ffd5684d6dffb1bb511142a2205f74aeb" Jul 10 17:25:39.710396 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.710339 2585 scope.go:117] "RemoveContainer" containerID="5d1f35696ac7dbb9ed3a12d30748d893cbbdbdbd6f52d7389411a2226242b4a1" Jul 10 17:25:39.719060 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.719033 2585 scope.go:117] "RemoveContainer" containerID="16d1416b69a167fe8106ae1432503c5ffd5684d6dffb1bb511142a2205f74aeb" Jul 10 17:25:39.719358 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:25:39.719339 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d1416b69a167fe8106ae1432503c5ffd5684d6dffb1bb511142a2205f74aeb\": container with ID starting with 16d1416b69a167fe8106ae1432503c5ffd5684d6dffb1bb511142a2205f74aeb not found: ID does not exist" containerID="16d1416b69a167fe8106ae1432503c5ffd5684d6dffb1bb511142a2205f74aeb" Jul 10 17:25:39.719414 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.719367 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d1416b69a167fe8106ae1432503c5ffd5684d6dffb1bb511142a2205f74aeb"} err="failed to get container status \"16d1416b69a167fe8106ae1432503c5ffd5684d6dffb1bb511142a2205f74aeb\": rpc error: code = NotFound desc = could not find container \"16d1416b69a167fe8106ae1432503c5ffd5684d6dffb1bb511142a2205f74aeb\": container with ID starting with 16d1416b69a167fe8106ae1432503c5ffd5684d6dffb1bb511142a2205f74aeb not found: ID does not exist" Jul 10 17:25:39.719414 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.719384 2585 scope.go:117] "RemoveContainer" containerID="5d1f35696ac7dbb9ed3a12d30748d893cbbdbdbd6f52d7389411a2226242b4a1" Jul 10 17:25:39.719609 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:25:39.719593 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1f35696ac7dbb9ed3a12d30748d893cbbdbdbd6f52d7389411a2226242b4a1\": container with ID starting with 5d1f35696ac7dbb9ed3a12d30748d893cbbdbdbd6f52d7389411a2226242b4a1 not found: ID does not exist" containerID="5d1f35696ac7dbb9ed3a12d30748d893cbbdbdbd6f52d7389411a2226242b4a1" Jul 10 17:25:39.719650 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.719615 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1f35696ac7dbb9ed3a12d30748d893cbbdbdbd6f52d7389411a2226242b4a1"} err="failed to get container status \"5d1f35696ac7dbb9ed3a12d30748d893cbbdbdbd6f52d7389411a2226242b4a1\": rpc error: code = NotFound desc = could not find container \"5d1f35696ac7dbb9ed3a12d30748d893cbbdbdbd6f52d7389411a2226242b4a1\": container with ID starting with 5d1f35696ac7dbb9ed3a12d30748d893cbbdbdbd6f52d7389411a2226242b4a1 not found: ID does not exist" Jul 10 17:25:39.721300 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.721280 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr"] Jul 10 17:25:39.727289 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:39.727268 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-fd7cfb-openshift-default-7f7f86df5f-2k8dr"] Jul 10 17:25:41.325986 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:41.325950 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07e2b632-14ce-43c9-89d8-7daae3e77154" path="/var/lib/kubelet/pods/07e2b632-14ce-43c9-89d8-7daae3e77154/volumes" Jul 10 17:25:47.695683 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:47.695646 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6nmhd"] Jul 10 17:25:59.788202 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:59.788167 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp"] Jul 10 17:25:59.788714 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:25:59.788429 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" podUID="c35dbe24-690a-4c79-a4da-007f9a351f09" containerName="istio-proxy" containerID="cri-o://f512e1eb6cf48cbec394707c117446bbd4e273ee8f589ad90361bb14d1421a94" gracePeriod=30 Jul 10 17:26:00.447400 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:00.447362 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-96d6c78b5-2tdp8_c64833da-5ba0-4a94-a050-88c6907421fa/authorino/0.log" Jul 10 17:26:02.595297 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:02.595258 2585 patch_prober.go:28] interesting pod/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 17:26:02.595666 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:02.595328 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" podUID="c35dbe24-690a-4c79-a4da-007f9a351f09" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 17:26:05.048982 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.048913 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:26:05.085141 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.085105 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-podinfo\") pod \"c35dbe24-690a-4c79-a4da-007f9a351f09\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " Jul 10 17:26:05.085141 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.085140 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-envoy\") pod \"c35dbe24-690a-4c79-a4da-007f9a351f09\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " Jul 10 17:26:05.085392 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.085165 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-token\") pod \"c35dbe24-690a-4c79-a4da-007f9a351f09\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " Jul 10 17:26:05.085392 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.085219 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-md5cf\" (UniqueName: \"kubernetes.io/projected/c35dbe24-690a-4c79-a4da-007f9a351f09-kube-api-access-md5cf\") pod \"c35dbe24-690a-4c79-a4da-007f9a351f09\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " Jul 10 17:26:05.085392 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.085252 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-data\") pod \"c35dbe24-690a-4c79-a4da-007f9a351f09\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " Jul 10 17:26:05.085537 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.085384 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35dbe24-690a-4c79-a4da-007f9a351f09-istiod-ca-cert\") pod \"c35dbe24-690a-4c79-a4da-007f9a351f09\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " Jul 10 17:26:05.085537 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.085460 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-credential-socket\") pod \"c35dbe24-690a-4c79-a4da-007f9a351f09\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " Jul 10 17:26:05.085537 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.085504 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-workload-socket\") pod \"c35dbe24-690a-4c79-a4da-007f9a351f09\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " Jul 10 17:26:05.085537 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.085528 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-workload-certs\") pod \"c35dbe24-690a-4c79-a4da-007f9a351f09\" (UID: \"c35dbe24-690a-4c79-a4da-007f9a351f09\") " Jul 10 17:26:05.085790 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.085763 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c35dbe24-690a-4c79-a4da-007f9a351f09" (UID: "c35dbe24-690a-4c79-a4da-007f9a351f09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:26:05.085851 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.085809 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35dbe24-690a-4c79-a4da-007f9a351f09-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c35dbe24-690a-4c79-a4da-007f9a351f09" (UID: "c35dbe24-690a-4c79-a4da-007f9a351f09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:26:05.086124 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.085958 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c35dbe24-690a-4c79-a4da-007f9a351f09" (UID: "c35dbe24-690a-4c79-a4da-007f9a351f09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:26:05.086124 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.086003 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c35dbe24-690a-4c79-a4da-007f9a351f09" (UID: "c35dbe24-690a-4c79-a4da-007f9a351f09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:26:05.086124 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.086062 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35dbe24-690a-4c79-a4da-007f9a351f09-istiod-ca-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:26:05.086124 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.086082 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-credential-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:26:05.086124 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.086096 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-workload-socket\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:26:05.086565 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.086494 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-data" (OuterVolumeSpecName: "istio-data") pod "c35dbe24-690a-4c79-a4da-007f9a351f09" (UID: "c35dbe24-690a-4c79-a4da-007f9a351f09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:26:05.087705 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.087655 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c35dbe24-690a-4c79-a4da-007f9a351f09" (UID: "c35dbe24-690a-4c79-a4da-007f9a351f09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:26:05.088098 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.088070 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35dbe24-690a-4c79-a4da-007f9a351f09-kube-api-access-md5cf" (OuterVolumeSpecName: "kube-api-access-md5cf") pod "c35dbe24-690a-4c79-a4da-007f9a351f09" (UID: "c35dbe24-690a-4c79-a4da-007f9a351f09"). InnerVolumeSpecName "kube-api-access-md5cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:26:05.088335 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.088318 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-token" (OuterVolumeSpecName: "istio-token") pod "c35dbe24-690a-4c79-a4da-007f9a351f09" (UID: "c35dbe24-690a-4c79-a4da-007f9a351f09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:26:05.088570 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.088553 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c35dbe24-690a-4c79-a4da-007f9a351f09" (UID: "c35dbe24-690a-4c79-a4da-007f9a351f09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:26:05.186736 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.186702 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-podinfo\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:26:05.186736 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.186732 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-envoy\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:26:05.186736 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.186740 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-token\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:26:05.187000 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.186770 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md5cf\" (UniqueName: \"kubernetes.io/projected/c35dbe24-690a-4c79-a4da-007f9a351f09-kube-api-access-md5cf\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:26:05.187000 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.186779 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-istio-data\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:26:05.187000 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.186789 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35dbe24-690a-4c79-a4da-007f9a351f09-workload-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\"" Jul 10 17:26:05.849645 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.849611 2585 generic.go:356] "Generic (PLEG): container finished" podID="c35dbe24-690a-4c79-a4da-007f9a351f09" containerID="f512e1eb6cf48cbec394707c117446bbd4e273ee8f589ad90361bb14d1421a94" exitCode=0 Jul 10 17:26:05.849841 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.849657 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" Jul 10 17:26:05.849841 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.849689 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" event={"ID":"c35dbe24-690a-4c79-a4da-007f9a351f09","Type":"ContainerDied","Data":"f512e1eb6cf48cbec394707c117446bbd4e273ee8f589ad90361bb14d1421a94"} Jul 10 17:26:05.849841 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.849762 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp" event={"ID":"c35dbe24-690a-4c79-a4da-007f9a351f09","Type":"ContainerDied","Data":"3e44aa5bcbbf4da3d1af564a0ffa65f174af45897e94a729e7b429fe974da05f"} Jul 10 17:26:05.849841 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.849790 2585 scope.go:117] "RemoveContainer" containerID="f512e1eb6cf48cbec394707c117446bbd4e273ee8f589ad90361bb14d1421a94" Jul 10 17:26:05.861444 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.861422 2585 scope.go:117] "RemoveContainer" containerID="0a4ce96218c18a2ebf99f0886fba9fdd168ef5d6fd5647e832b3da4a22a34998" Jul 10 17:26:05.869753 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.869723 2585 scope.go:117] "RemoveContainer" containerID="f512e1eb6cf48cbec394707c117446bbd4e273ee8f589ad90361bb14d1421a94" Jul 10 17:26:05.870013 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:26:05.869981 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f512e1eb6cf48cbec394707c117446bbd4e273ee8f589ad90361bb14d1421a94\": container with ID starting with f512e1eb6cf48cbec394707c117446bbd4e273ee8f589ad90361bb14d1421a94 not found: ID does not exist" containerID="f512e1eb6cf48cbec394707c117446bbd4e273ee8f589ad90361bb14d1421a94" Jul 10 17:26:05.870099 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.870021 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f512e1eb6cf48cbec394707c117446bbd4e273ee8f589ad90361bb14d1421a94"} err="failed to get container status \"f512e1eb6cf48cbec394707c117446bbd4e273ee8f589ad90361bb14d1421a94\": rpc error: code = NotFound desc = could not find container \"f512e1eb6cf48cbec394707c117446bbd4e273ee8f589ad90361bb14d1421a94\": container with ID starting with f512e1eb6cf48cbec394707c117446bbd4e273ee8f589ad90361bb14d1421a94 not found: ID does not exist" Jul 10 17:26:05.870099 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.870037 2585 scope.go:117] "RemoveContainer" containerID="0a4ce96218c18a2ebf99f0886fba9fdd168ef5d6fd5647e832b3da4a22a34998" Jul 10 17:26:05.870247 ip-10-0-134-41 kubenswrapper[2585]: E0710 17:26:05.870229 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a4ce96218c18a2ebf99f0886fba9fdd168ef5d6fd5647e832b3da4a22a34998\": container with ID starting with 0a4ce96218c18a2ebf99f0886fba9fdd168ef5d6fd5647e832b3da4a22a34998 not found: ID does not exist" containerID="0a4ce96218c18a2ebf99f0886fba9fdd168ef5d6fd5647e832b3da4a22a34998" Jul 10 17:26:05.870289 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.870253 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4ce96218c18a2ebf99f0886fba9fdd168ef5d6fd5647e832b3da4a22a34998"} err="failed to get container status \"0a4ce96218c18a2ebf99f0886fba9fdd168ef5d6fd5647e832b3da4a22a34998\": rpc error: code = NotFound desc = could not find container \"0a4ce96218c18a2ebf99f0886fba9fdd168ef5d6fd5647e832b3da4a22a34998\": container with ID starting with 0a4ce96218c18a2ebf99f0886fba9fdd168ef5d6fd5647e832b3da4a22a34998 not found: ID does not exist" Jul 10 17:26:05.882578 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.882551 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp"] Jul 10 17:26:05.887708 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:05.887684 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-c9789a-openshift-default-55568f5c79-d4tnp"] Jul 10 17:26:06.339311 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:06.339282 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-585db49996-v6pvq_ea27ada3-c6a4-4a07-947d-f92f52af0bb0/copy-manifests/0.log" Jul 10 17:26:06.354338 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:06.354308 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-585db49996-v6pvq_ea27ada3-c6a4-4a07-947d-f92f52af0bb0/manager/0.log" Jul 10 17:26:07.325902 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:07.325868 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c35dbe24-690a-4c79-a4da-007f9a351f09" path="/var/lib/kubelet/pods/c35dbe24-690a-4c79-a4da-007f9a351f09/volumes" Jul 10 17:26:09.108108 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:09.108071 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx_60523f32-9f6f-4937-99c2-fc7aa9d96655/pull/0.log" Jul 10 17:26:09.114325 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:09.114301 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx_60523f32-9f6f-4937-99c2-fc7aa9d96655/extract/0.log" Jul 10 17:26:09.120107 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:09.120073 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx_60523f32-9f6f-4937-99c2-fc7aa9d96655/util/0.log" Jul 10 17:26:09.230138 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:09.230107 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45_b235db81-1b87-4a7c-8a5a-547ce8ca845a/extract/0.log" Jul 10 17:26:09.236668 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:09.236641 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45_b235db81-1b87-4a7c-8a5a-547ce8ca845a/util/0.log" Jul 10 17:26:09.244282 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:09.244261 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45_b235db81-1b87-4a7c-8a5a-547ce8ca845a/pull/0.log" Jul 10 17:26:09.350560 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:09.350529 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv_c984eaa6-2d5a-41e1-8b6c-1def3e97c81a/extract/0.log" Jul 10 17:26:09.357217 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:09.357186 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv_c984eaa6-2d5a-41e1-8b6c-1def3e97c81a/util/0.log" Jul 10 17:26:09.364073 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:09.364019 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv_c984eaa6-2d5a-41e1-8b6c-1def3e97c81a/pull/0.log" Jul 10 17:26:09.555570 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:09.555543 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h_1f997c02-2eaa-4bc0-8a27-e391e6530f64/util/0.log" Jul 10 17:26:09.561996 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:09.561970 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h_1f997c02-2eaa-4bc0-8a27-e391e6530f64/pull/0.log" Jul 10 17:26:09.568296 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:09.568269 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h_1f997c02-2eaa-4bc0-8a27-e391e6530f64/extract/0.log" Jul 10 17:26:09.698012 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:09.697929 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-96d6c78b5-2tdp8_c64833da-5ba0-4a94-a050-88c6907421fa/authorino/0.log" Jul 10 17:26:09.926125 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:09.926099 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5l6zf_f23fb5ac-7e1f-4fd4-99bc-b368b3204ff0/manager/0.log" Jul 10 17:26:10.035492 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:10.035403 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-sgsw7_a1780a33-28d0-4f8c-9388-91eba6f0c996/kuadrant-console-plugin/0.log" Jul 10 17:26:10.153826 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:10.153794 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-sn6r9_2958cd40-3889-4149-86ff-70d50b439824/registry-server/0.log" Jul 10 17:26:10.462363 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:10.462330 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-6nmhd_4b26840e-4e20-4666-bacc-6deb8f4e0f37/limitador/0.log" Jul 10 17:26:10.576053 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:10.576024 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-4fkhd_52147cde-c5c4-4dcb-9a5b-f1ec2d81efa2/manager/0.log" Jul 10 17:26:10.930217 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:10.930188 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hdmv8_d0909b02-35a0-4e85-9c1f-a43508d37f0a/istio-proxy/0.log" Jul 10 17:26:11.410585 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:11.410554 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cvchv_00adb9cb-ed88-4eb5-bed7-63f59639ae86/istio-proxy/1.log" Jul 10 17:26:11.777350 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:11.777265 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b9b6c86cb-knlg6_6a5712a8-cdf6-4acc-afb0-21827ce7bbf4/router/0.log" Jul 10 17:26:12.679002 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:12.678968 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-tvf29_356f50aa-3017-49a8-8733-c4dfa7ec8ee7/main/0.log" Jul 10 17:26:16.951738 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.951700 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bq8tx/must-gather-prpmg"] Jul 10 17:26:16.952269 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.952251 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c35dbe24-690a-4c79-a4da-007f9a351f09" containerName="istio-proxy" Jul 10 17:26:16.952354 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.952272 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c35dbe24-690a-4c79-a4da-007f9a351f09" containerName="istio-proxy" Jul 10 17:26:16.952354 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.952284 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa694c7-5602-432a-8a14-64e32f09197b" containerName="test-curl-7504-139898144300896" Jul 10 17:26:16.952354 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.952292 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa694c7-5602-432a-8a14-64e32f09197b" containerName="test-curl-7504-139898144300896" Jul 10 17:26:16.952354 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.952306 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e2b632-14ce-43c9-89d8-7daae3e77154" containerName="istio-proxy" Jul 10 17:26:16.952354 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.952315 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e2b632-14ce-43c9-89d8-7daae3e77154" containerName="istio-proxy" Jul 10 17:26:16.952354 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.952323 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e2b632-14ce-43c9-89d8-7daae3e77154" containerName="istio-proxy" Jul 10 17:26:16.952354 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.952330 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e2b632-14ce-43c9-89d8-7daae3e77154" containerName="istio-proxy" Jul 10 17:26:16.952354 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.952349 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c35dbe24-690a-4c79-a4da-007f9a351f09" containerName="istio-proxy" Jul 10 17:26:16.952788 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.952359 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c35dbe24-690a-4c79-a4da-007f9a351f09" containerName="istio-proxy" Jul 10 17:26:16.952788 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.952447 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c35dbe24-690a-4c79-a4da-007f9a351f09" containerName="istio-proxy" Jul 10 17:26:16.952788 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.952458 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aa694c7-5602-432a-8a14-64e32f09197b" containerName="test-curl-7504-139898144300896" Jul 10 17:26:16.952788 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.952466 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e2b632-14ce-43c9-89d8-7daae3e77154" containerName="istio-proxy" Jul 10 17:26:16.952788 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.952477 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e2b632-14ce-43c9-89d8-7daae3e77154" containerName="istio-proxy" Jul 10 17:26:16.952788 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.952487 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c35dbe24-690a-4c79-a4da-007f9a351f09" containerName="istio-proxy" Jul 10 17:26:16.957120 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.957098 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq8tx/must-gather-prpmg" Jul 10 17:26:16.959951 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.959931 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bq8tx\"/\"default-dockercfg-w74x2\"" Jul 10 17:26:16.960485 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.960455 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bq8tx\"/\"kube-root-ca.crt\"" Jul 10 17:26:16.960601 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.960533 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bq8tx\"/\"openshift-service-ca.crt\"" Jul 10 17:26:16.966089 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:16.966071 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq8tx/must-gather-prpmg"] Jul 10 17:26:17.095844 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:17.095820 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d206a5cb-aa20-48ad-bf99-bd2acd4d67df-must-gather-output\") pod \"must-gather-prpmg\" (UID: \"d206a5cb-aa20-48ad-bf99-bd2acd4d67df\") " pod="openshift-must-gather-bq8tx/must-gather-prpmg" Jul 10 17:26:17.095963 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:17.095851 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bx8b\" (UniqueName: \"kubernetes.io/projected/d206a5cb-aa20-48ad-bf99-bd2acd4d67df-kube-api-access-2bx8b\") pod \"must-gather-prpmg\" (UID: \"d206a5cb-aa20-48ad-bf99-bd2acd4d67df\") " pod="openshift-must-gather-bq8tx/must-gather-prpmg" Jul 10 17:26:17.196175 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:17.196151 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d206a5cb-aa20-48ad-bf99-bd2acd4d67df-must-gather-output\") pod \"must-gather-prpmg\" (UID: \"d206a5cb-aa20-48ad-bf99-bd2acd4d67df\") " pod="openshift-must-gather-bq8tx/must-gather-prpmg" Jul 10 17:26:17.196272 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:17.196180 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bx8b\" (UniqueName: \"kubernetes.io/projected/d206a5cb-aa20-48ad-bf99-bd2acd4d67df-kube-api-access-2bx8b\") pod \"must-gather-prpmg\" (UID: \"d206a5cb-aa20-48ad-bf99-bd2acd4d67df\") " pod="openshift-must-gather-bq8tx/must-gather-prpmg" Jul 10 17:26:17.196452 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:17.196434 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d206a5cb-aa20-48ad-bf99-bd2acd4d67df-must-gather-output\") pod \"must-gather-prpmg\" (UID: \"d206a5cb-aa20-48ad-bf99-bd2acd4d67df\") " pod="openshift-must-gather-bq8tx/must-gather-prpmg" Jul 10 17:26:17.212938 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:17.212885 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bx8b\" (UniqueName: \"kubernetes.io/projected/d206a5cb-aa20-48ad-bf99-bd2acd4d67df-kube-api-access-2bx8b\") pod \"must-gather-prpmg\" (UID: \"d206a5cb-aa20-48ad-bf99-bd2acd4d67df\") " pod="openshift-must-gather-bq8tx/must-gather-prpmg" Jul 10 17:26:17.267884 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:17.267863 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq8tx/must-gather-prpmg" Jul 10 17:26:17.402000 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:17.401973 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq8tx/must-gather-prpmg"] Jul 10 17:26:17.403355 ip-10-0-134-41 kubenswrapper[2585]: W0710 17:26:17.403329 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd206a5cb_aa20_48ad_bf99_bd2acd4d67df.slice/crio-378466c048d9ef8627c6d14412f1fc2d610b5d19d2036a3eb6ec533bfef09558 WatchSource:0}: Error finding container 378466c048d9ef8627c6d14412f1fc2d610b5d19d2036a3eb6ec533bfef09558: Status 404 returned error can't find the container with id 378466c048d9ef8627c6d14412f1fc2d610b5d19d2036a3eb6ec533bfef09558 Jul 10 17:26:17.914812 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:17.914780 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq8tx/must-gather-prpmg" event={"ID":"d206a5cb-aa20-48ad-bf99-bd2acd4d67df","Type":"ContainerStarted","Data":"378466c048d9ef8627c6d14412f1fc2d610b5d19d2036a3eb6ec533bfef09558"} Jul 10 17:26:18.925114 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:18.924467 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq8tx/must-gather-prpmg" event={"ID":"d206a5cb-aa20-48ad-bf99-bd2acd4d67df","Type":"ContainerStarted","Data":"3cc426afa9bb0c48b7b6b83bec98b94fea8eedfae2c2322f49fbbaff7a8b72ac"} Jul 10 17:26:18.925114 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:18.924525 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq8tx/must-gather-prpmg" event={"ID":"d206a5cb-aa20-48ad-bf99-bd2acd4d67df","Type":"ContainerStarted","Data":"7eaf7c3fefdc22baa3acdc5fcfe7161ccef4e410e7f6d8bfa10fa544570d3904"} Jul 10 17:26:18.947763 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:18.944270 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bq8tx/must-gather-prpmg" podStartSLOduration=2.099859291 podStartE2EDuration="2.944250831s" totalImagesPullingTime="844.39154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:26:18.940901809 +0000 UTC m=+2998.362778278" watchObservedRunningTime="2026-07-10 17:26:18.944250831 +0000 UTC m=+2998.366127299" Jul 10 17:26:19.825474 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:19.825394 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lhp9w_2594870a-c2d7-47e9-8f55-d6c029810a1d/global-pull-secret-syncer/0.log" Jul 10 17:26:19.986343 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:19.986312 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nzk6t_60be53b8-7bc0-456c-9029-76e0f65316e6/konnectivity-agent/0.log" Jul 10 17:26:20.076542 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:20.076447 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-41.ec2.internal_e404ee0301f15458abcdaff3158e6ab9/haproxy/0.log" Jul 10 17:26:21.345565 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:21.345531 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 17:26:21.353474 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:21.353442 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lqv6_b4e0a636-8e9b-42fb-bfb6-b4bc543a4a56/ovn-acl-logging/0.log" Jul 10 17:26:23.742102 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:23.742059 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx_60523f32-9f6f-4937-99c2-fc7aa9d96655/extract/0.log" Jul 10 17:26:23.776921 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:23.776892 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx_60523f32-9f6f-4937-99c2-fc7aa9d96655/util/0.log" Jul 10 17:26:23.799427 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:23.799384 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75986ldx_60523f32-9f6f-4937-99c2-fc7aa9d96655/pull/0.log" Jul 10 17:26:23.826809 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:23.826774 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45_b235db81-1b87-4a7c-8a5a-547ce8ca845a/extract/0.log" Jul 10 17:26:23.851760 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:23.851707 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45_b235db81-1b87-4a7c-8a5a-547ce8ca845a/util/0.log" Jul 10 17:26:23.876014 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:23.875988 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0brv45_b235db81-1b87-4a7c-8a5a-547ce8ca845a/pull/0.log" Jul 10 17:26:23.917540 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:23.917506 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv_c984eaa6-2d5a-41e1-8b6c-1def3e97c81a/extract/0.log" Jul 10 17:26:23.944153 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:23.944119 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv_c984eaa6-2d5a-41e1-8b6c-1def3e97c81a/util/0.log" Jul 10 17:26:23.971525 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:23.971493 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xjjpv_c984eaa6-2d5a-41e1-8b6c-1def3e97c81a/pull/0.log" Jul 10 17:26:23.999348 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:23.999270 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h_1f997c02-2eaa-4bc0-8a27-e391e6530f64/extract/0.log" Jul 10 17:26:24.032523 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:24.032490 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h_1f997c02-2eaa-4bc0-8a27-e391e6530f64/util/0.log" Jul 10 17:26:24.059156 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:24.059096 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fp79h_1f997c02-2eaa-4bc0-8a27-e391e6530f64/pull/0.log" Jul 10 17:26:24.447769 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:24.447394 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-96d6c78b5-2tdp8_c64833da-5ba0-4a94-a050-88c6907421fa/authorino/0.log" Jul 10 17:26:24.504332 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:24.504301 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5l6zf_f23fb5ac-7e1f-4fd4-99bc-b368b3204ff0/manager/0.log" Jul 10 17:26:24.531261 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:24.531217 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-sgsw7_a1780a33-28d0-4f8c-9388-91eba6f0c996/kuadrant-console-plugin/0.log" Jul 10 17:26:24.575731 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:24.575654 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-sn6r9_2958cd40-3889-4149-86ff-70d50b439824/registry-server/0.log" Jul 10 17:26:24.811602 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:24.811567 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-6nmhd_4b26840e-4e20-4666-bacc-6deb8f4e0f37/limitador/0.log" Jul 10 17:26:24.893897 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:24.893860 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-4fkhd_52147cde-c5c4-4dcb-9a5b-f1ec2d81efa2/manager/0.log" Jul 10 17:26:26.494553 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:26.494504 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_72e7ded4-2f33-4ac5-918b-04a2caee91e1/alertmanager/0.log" Jul 10 17:26:26.525153 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:26.525115 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_72e7ded4-2f33-4ac5-918b-04a2caee91e1/config-reloader/0.log" Jul 10 17:26:26.548931 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:26.548885 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_72e7ded4-2f33-4ac5-918b-04a2caee91e1/kube-rbac-proxy-web/0.log" Jul 10 17:26:26.575060 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:26.575035 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_72e7ded4-2f33-4ac5-918b-04a2caee91e1/kube-rbac-proxy/0.log" Jul 10 17:26:26.598140 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:26.598109 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_72e7ded4-2f33-4ac5-918b-04a2caee91e1/kube-rbac-proxy-metric/0.log" Jul 10 17:26:26.622994 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:26.622966 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_72e7ded4-2f33-4ac5-918b-04a2caee91e1/prom-label-proxy/0.log" Jul 10 17:26:26.647894 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:26.647830 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_72e7ded4-2f33-4ac5-918b-04a2caee91e1/init-config-reloader/0.log" Jul 10 17:26:26.725521 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:26.725480 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-nkpt6_f425e231-4c8e-4482-b2cc-bec2266b9a93/kube-state-metrics/0.log" Jul 10 17:26:26.748116 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:26.748030 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-nkpt6_f425e231-4c8e-4482-b2cc-bec2266b9a93/kube-rbac-proxy-main/0.log" Jul 10 17:26:26.772802 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:26.772770 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-nkpt6_f425e231-4c8e-4482-b2cc-bec2266b9a93/kube-rbac-proxy-self/0.log" Jul 10 17:26:26.809273 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:26.809241 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-d7c4f5f87-jx56r_60146d46-3d75-41de-8d5a-3dba67f81a88/metrics-server/0.log" Jul 10 17:26:26.861438 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:26.861381 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6tmzw_5eecaa1e-89cc-4bb3-97bf-05d9d5d32683/node-exporter/0.log" Jul 10 17:26:26.886010 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:26.885978 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6tmzw_5eecaa1e-89cc-4bb3-97bf-05d9d5d32683/kube-rbac-proxy/0.log" Jul 10 17:26:26.908922 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:26.908892 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6tmzw_5eecaa1e-89cc-4bb3-97bf-05d9d5d32683/init-textfile/0.log" Jul 10 17:26:27.134072 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:27.134035 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-n6hcx_73e50952-6b77-4e60-a941-91629e495158/kube-rbac-proxy-main/0.log" Jul 10 17:26:27.155999 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:27.155966 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-n6hcx_73e50952-6b77-4e60-a941-91629e495158/kube-rbac-proxy-self/0.log" Jul 10 17:26:27.187234 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:27.187170 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-n6hcx_73e50952-6b77-4e60-a941-91629e495158/openshift-state-metrics/0.log" Jul 10 17:26:27.468291 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:27.468189 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-qr7rl_d394d15a-75b8-4f95-9bd3-06aaa8b9d6c3/prometheus-operator-admission-webhook/0.log" Jul 10 17:26:30.167882 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.167852 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-54cd88dfbf-sw796_04deca7a-3598-40c3-9fe6-c247f1712840/console/0.log" Jul 10 17:26:30.195550 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.195514 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs"] Jul 10 17:26:30.200430 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.200405 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:30.209051 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.209013 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs"] Jul 10 17:26:30.348410 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.348378 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e171be1f-26b4-4730-9fa3-a398130dd764-sys\") pod \"perf-node-gather-daemonset-g7ncs\" (UID: \"e171be1f-26b4-4730-9fa3-a398130dd764\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:30.348601 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.348452 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e171be1f-26b4-4730-9fa3-a398130dd764-podres\") pod \"perf-node-gather-daemonset-g7ncs\" (UID: \"e171be1f-26b4-4730-9fa3-a398130dd764\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:30.348651 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.348605 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf4j8\" (UniqueName: \"kubernetes.io/projected/e171be1f-26b4-4730-9fa3-a398130dd764-kube-api-access-pf4j8\") pod \"perf-node-gather-daemonset-g7ncs\" (UID: \"e171be1f-26b4-4730-9fa3-a398130dd764\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:30.348698 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.348656 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e171be1f-26b4-4730-9fa3-a398130dd764-proc\") pod \"perf-node-gather-daemonset-g7ncs\" (UID: \"e171be1f-26b4-4730-9fa3-a398130dd764\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:30.348698 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.348682 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e171be1f-26b4-4730-9fa3-a398130dd764-lib-modules\") pod \"perf-node-gather-daemonset-g7ncs\" (UID: \"e171be1f-26b4-4730-9fa3-a398130dd764\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:30.450164 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.450080 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf4j8\" (UniqueName: \"kubernetes.io/projected/e171be1f-26b4-4730-9fa3-a398130dd764-kube-api-access-pf4j8\") pod \"perf-node-gather-daemonset-g7ncs\" (UID: \"e171be1f-26b4-4730-9fa3-a398130dd764\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:30.450164 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.450135 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e171be1f-26b4-4730-9fa3-a398130dd764-proc\") pod \"perf-node-gather-daemonset-g7ncs\" (UID: \"e171be1f-26b4-4730-9fa3-a398130dd764\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:30.450164 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.450154 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e171be1f-26b4-4730-9fa3-a398130dd764-lib-modules\") pod \"perf-node-gather-daemonset-g7ncs\" (UID: \"e171be1f-26b4-4730-9fa3-a398130dd764\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:30.450446 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.450212 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e171be1f-26b4-4730-9fa3-a398130dd764-sys\") pod \"perf-node-gather-daemonset-g7ncs\" (UID: \"e171be1f-26b4-4730-9fa3-a398130dd764\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:30.450446 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.450215 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e171be1f-26b4-4730-9fa3-a398130dd764-proc\") pod \"perf-node-gather-daemonset-g7ncs\" (UID: \"e171be1f-26b4-4730-9fa3-a398130dd764\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:30.450446 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.450270 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e171be1f-26b4-4730-9fa3-a398130dd764-sys\") pod \"perf-node-gather-daemonset-g7ncs\" (UID: \"e171be1f-26b4-4730-9fa3-a398130dd764\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:30.450446 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.450284 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e171be1f-26b4-4730-9fa3-a398130dd764-lib-modules\") pod \"perf-node-gather-daemonset-g7ncs\" (UID: \"e171be1f-26b4-4730-9fa3-a398130dd764\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:30.450446 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.450321 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e171be1f-26b4-4730-9fa3-a398130dd764-podres\") pod \"perf-node-gather-daemonset-g7ncs\" (UID: \"e171be1f-26b4-4730-9fa3-a398130dd764\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:30.450446 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.450425 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e171be1f-26b4-4730-9fa3-a398130dd764-podres\") pod \"perf-node-gather-daemonset-g7ncs\" (UID: \"e171be1f-26b4-4730-9fa3-a398130dd764\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:30.458491 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.458462 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf4j8\" (UniqueName: \"kubernetes.io/projected/e171be1f-26b4-4730-9fa3-a398130dd764-kube-api-access-pf4j8\") pod \"perf-node-gather-daemonset-g7ncs\" (UID: \"e171be1f-26b4-4730-9fa3-a398130dd764\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:30.513683 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.513646 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:30.683799 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:30.683759 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs"] Jul 10 17:26:31.026596 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:31.026500 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" event={"ID":"e171be1f-26b4-4730-9fa3-a398130dd764","Type":"ContainerStarted","Data":"76915d522a6fe33ab7b4a7548f6ab360984582dde22e320366425c8caf1cfcec"} Jul 10 17:26:31.026799 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:31.026600 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" event={"ID":"e171be1f-26b4-4730-9fa3-a398130dd764","Type":"ContainerStarted","Data":"dc288dbc6882c13d01c249ba6fcae9b558831b0a5740d2f5ff1395ba709ebdfa"} Jul 10 17:26:31.026904 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:31.026884 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:31.043245 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:31.043190 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" podStartSLOduration=1.043173411 podStartE2EDuration="1.043173411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:26:31.041522772 +0000 UTC m=+3010.463399241" watchObservedRunningTime="2026-07-10 17:26:31.043173411 +0000 UTC m=+3010.465049877" Jul 10 17:26:32.011783 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:32.011756 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kp9dv_32d1ff51-842a-4cb2-b179-ffe84520aad3/dns/0.log" Jul 10 17:26:32.035141 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:32.035109 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kp9dv_32d1ff51-842a-4cb2-b179-ffe84520aad3/kube-rbac-proxy/0.log" Jul 10 17:26:32.115952 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:32.115924 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m7vs8_2d492a8c-19e2-462a-85bc-ac78c1898d73/dns-node-resolver/0.log" Jul 10 17:26:32.678064 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:32.678016 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-77d699fcbb-vqkjx_a725bedd-e82d-4abd-b3f6-888d01da0fc4/registry/0.log" Jul 10 17:26:32.729193 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:32.729168 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dsv6h_1d54c0b4-7530-45fb-804c-777c15ff35eb/node-ca/0.log" Jul 10 17:26:33.697594 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:33.697561 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hdmv8_d0909b02-35a0-4e85-9c1f-a43508d37f0a/istio-proxy/0.log" Jul 10 17:26:33.982872 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:33.982797 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cvchv_00adb9cb-ed88-4eb5-bed7-63f59639ae86/istio-proxy/0.log" Jul 10 17:26:34.019203 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:34.019168 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cvchv_00adb9cb-ed88-4eb5-bed7-63f59639ae86/istio-proxy/1.log" Jul 10 17:26:34.109859 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:34.109831 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b9b6c86cb-knlg6_6a5712a8-cdf6-4acc-afb0-21827ce7bbf4/router/0.log" Jul 10 17:26:34.807342 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:34.807313 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hbr2f_7ea783bf-0092-4036-a31b-bc4111a252c5/serve-healthcheck-canary/0.log" Jul 10 17:26:35.372820 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:35.372790 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4bncj_8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d/kube-rbac-proxy/0.log" Jul 10 17:26:35.397362 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:35.397328 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4bncj_8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d/exporter/0.log" Jul 10 17:26:35.421666 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:35.421641 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4bncj_8d9f9cd0-98a6-4deb-8032-e0c4f9bbbc9d/extractor/0.log" Jul 10 17:26:37.043102 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:37.043072 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-g7ncs" Jul 10 17:26:38.395597 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:38.395566 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-585db49996-v6pvq_ea27ada3-c6a4-4a07-947d-f92f52af0bb0/manager/0.log" Jul 10 17:26:38.414258 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:38.414234 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-585db49996-v6pvq_ea27ada3-c6a4-4a07-947d-f92f52af0bb0/copy-manifests/0.log" Jul 10 17:26:39.778046 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:39.778009 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-655c89f56d-g5rqd_990c9d67-17b9-4bda-a748-b4676da25ff0/manager/0.log" Jul 10 17:26:39.917258 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:39.917222 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hdmv8_d0909b02-35a0-4e85-9c1f-a43508d37f0a/istio-proxy/0.log" Jul 10 17:26:40.191828 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:40.191727 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cvchv_00adb9cb-ed88-4eb5-bed7-63f59639ae86/istio-proxy/0.log" Jul 10 17:26:40.229196 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:40.228723 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cvchv_00adb9cb-ed88-4eb5-bed7-63f59639ae86/istio-proxy/1.log" Jul 10 17:26:40.306567 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:40.306535 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b9b6c86cb-knlg6_6a5712a8-cdf6-4acc-afb0-21827ce7bbf4/router/0.log" Jul 10 17:26:41.604937 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:41.604909 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hdmv8_d0909b02-35a0-4e85-9c1f-a43508d37f0a/istio-proxy/0.log" Jul 10 17:26:41.852837 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:41.852795 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cvchv_00adb9cb-ed88-4eb5-bed7-63f59639ae86/istio-proxy/0.log" Jul 10 17:26:41.891802 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:41.891700 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cvchv_00adb9cb-ed88-4eb5-bed7-63f59639ae86/istio-proxy/1.log" Jul 10 17:26:41.945761 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:41.945717 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b9b6c86cb-knlg6_6a5712a8-cdf6-4acc-afb0-21827ce7bbf4/router/0.log" Jul 10 17:26:45.325225 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:45.325141 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-jf7sf_f6f755a9-cc53-4a26-a3ea-8054a806bf74/kube-storage-version-migrator-operator/1.log" Jul 10 17:26:45.326148 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:45.326119 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-jf7sf_f6f755a9-cc53-4a26-a3ea-8054a806bf74/kube-storage-version-migrator-operator/0.log" Jul 10 17:26:46.315671 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:46.315637 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fnqd8_4a75db21-5ca1-43b4-83f7-57bfe805b360/kube-multus-additional-cni-plugins/0.log" Jul 10 17:26:46.339975 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:46.339943 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fnqd8_4a75db21-5ca1-43b4-83f7-57bfe805b360/egress-router-binary-copy/0.log" Jul 10 17:26:46.366361 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:46.366336 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fnqd8_4a75db21-5ca1-43b4-83f7-57bfe805b360/cni-plugins/0.log" Jul 10 17:26:46.388454 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:46.388431 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fnqd8_4a75db21-5ca1-43b4-83f7-57bfe805b360/bond-cni-plugin/0.log" Jul 10 17:26:46.410517 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:46.410491 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fnqd8_4a75db21-5ca1-43b4-83f7-57bfe805b360/routeoverride-cni/0.log" Jul 10 17:26:46.432671 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:46.432647 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fnqd8_4a75db21-5ca1-43b4-83f7-57bfe805b360/whereabouts-cni-bincopy/0.log" Jul 10 17:26:46.456279 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:46.456258 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fnqd8_4a75db21-5ca1-43b4-83f7-57bfe805b360/whereabouts-cni/0.log" Jul 10 17:26:46.825884 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:46.825851 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fmt8m_405fff59-3355-4cd2-9371-77c04c509283/kube-multus/0.log" Jul 10 17:26:46.920181 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:46.920136 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6c49f_08a4d929-2c36-40ac-a3e6-8c5caf0c1788/network-metrics-daemon/0.log" Jul 10 17:26:46.940825 ip-10-0-134-41 kubenswrapper[2585]: I0710 17:26:46.940800 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6c49f_08a4d929-2c36-40ac-a3e6-8c5caf0c1788/kube-rbac-proxy/0.log"