Jul 10 13:48:12.306885 ip-10-0-143-25 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 13:48:12.306900 ip-10-0-143-25 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 13:48:12.306910 ip-10-0-143-25 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 13:48:12.307229 ip-10-0-143-25 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 13:48:21.820189 ip-10-0-143-25 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Jul 10 13:48:21.820200 ip-10-0-143-25 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 1eb08491b698402a80e095bee599823b -- Jul 10 13:49:52.414573 ip-10-0-143-25 systemd[1]: Starting Kubernetes Kubelet... Jul 10 13:49:52.838200 ip-10-0-143-25 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 13:49:52.838200 ip-10-0-143-25 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 13:49:52.838200 ip-10-0-143-25 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 13:49:52.838200 ip-10-0-143-25 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 13:49:52.838200 ip-10-0-143-25 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 13:49:52.838954 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.838266 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 13:49:52.844600 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844574 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 13:49:52.844600 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844598 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844604 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844607 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844611 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844614 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844617 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844619 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844622 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844625 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844628 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844630 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844633 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844636 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844638 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844642 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844647 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844650 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844653 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844655 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844658 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 13:49:52.844688 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844660 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844663 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844666 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844668 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844671 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844673 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844676 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844679 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844683 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844687 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844690 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844694 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844697 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844700 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844703 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844706 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844708 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844711 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844714 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844717 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 13:49:52.845180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844719 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844722 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844724 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844727 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844729 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844732 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844735 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844739 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844743 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844746 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844749 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844751 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844754 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844756 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844761 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844763 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844766 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844768 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844771 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 13:49:52.845691 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844774 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844776 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844778 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844781 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844784 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844786 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844789 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844791 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844794 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844797 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844800 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844803 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844818 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844821 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844824 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844827 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844829 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844834 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844837 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844841 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 13:49:52.846162 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844844 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844847 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844850 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844853 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844856 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844859 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844862 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844866 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844870 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844874 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844877 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844880 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844883 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844885 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844888 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844890 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844893 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844895 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.844898 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845648 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 13:49:52.846638 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845655 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845658 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845661 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845664 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845667 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845670 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845672 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845675 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845678 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845680 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845683 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845690 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845694 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845697 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845700 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845703 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845707 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845710 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845713 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845716 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 13:49:52.847141 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845718 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845721 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845724 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845727 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845730 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845732 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845735 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845737 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845740 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845742 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845745 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845747 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845750 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845752 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845755 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845757 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845760 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845763 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845766 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845768 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 13:49:52.847652 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845771 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845773 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845776 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845779 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845782 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845784 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845787 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845789 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845792 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845794 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845797 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845799 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845802 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845821 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845825 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845828 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845831 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845834 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845837 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 13:49:52.848175 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845840 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845843 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845846 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845849 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845851 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845854 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845858 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845860 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845865 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845868 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845871 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845874 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845877 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845879 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845882 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845885 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845889 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845891 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845894 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 13:49:52.848653 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845897 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845899 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845902 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845904 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845907 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845910 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845912 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845915 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845917 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845920 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845923 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845925 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845927 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845930 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845932 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845935 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845938 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845940 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845943 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.845945 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 13:49:52.849143 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846386 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846391 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846395 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846397 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846401 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846404 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846408 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846412 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846415 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846419 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846421 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846424 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846427 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846430 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846432 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846435 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846438 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846442 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846445 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 13:49:52.849733 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846448 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846450 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846453 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846455 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846458 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846460 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846463 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846466 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846468 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846471 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846473 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846476 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846478 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846481 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846483 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846486 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846488 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846490 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846494 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846496 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 13:49:52.850221 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846499 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846501 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846505 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846507 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846510 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846513 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846515 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846517 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846520 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846522 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846525 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846528 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846530 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846533 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846535 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846538 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846541 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846544 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846546 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 13:49:52.850689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846548 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846551 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846554 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846556 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846559 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846561 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846563 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846566 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846569 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846571 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846573 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846576 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846579 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846581 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846584 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846587 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846589 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846592 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846595 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846597 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 13:49:52.851199 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846599 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846602 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846605 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846607 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846611 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846613 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846616 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846618 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846622 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846625 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846628 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846630 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846633 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846635 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846638 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846640 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846643 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846646 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846649 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 13:49:52.851689 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846651 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.846654 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846725 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846732 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846739 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846743 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846748 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846751 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846756 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846761 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846767 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846770 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846774 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846777 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846781 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846784 2576 flags.go:64] FLAG: --cgroup-root="" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846786 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846790 2576 flags.go:64] FLAG: --client-ca-file="" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846793 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846796 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846801 2576 flags.go:64] FLAG: --cluster-domain="" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846816 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846820 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846823 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 13:49:52.852184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846827 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846834 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846837 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846840 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846843 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846847 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846851 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846853 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846857 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846860 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846864 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846867 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846869 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846872 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846875 2576 flags.go:64] FLAG: --enable-server="true" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846879 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846883 2576 flags.go:64] FLAG: --event-burst="100" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846887 2576 flags.go:64] FLAG: --event-qps="50" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846895 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846898 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846901 2576 flags.go:64] FLAG: --eviction-hard="" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846905 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846908 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846911 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846914 2576 flags.go:64] FLAG: --eviction-soft="" Jul 10 13:49:52.852791 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846917 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846920 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846922 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846925 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846928 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846931 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846934 2576 flags.go:64] FLAG: --feature-gates="" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846938 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846941 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846944 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846947 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846950 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846953 2576 flags.go:64] FLAG: --help="false" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846956 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-143-25.ec2.internal" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846958 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846961 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846964 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846970 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846974 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846976 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846979 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846982 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846985 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 13:49:52.853400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846989 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846992 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.846997 2576 flags.go:64] FLAG: --kube-reserved="" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847000 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847011 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847015 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847018 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847021 2576 flags.go:64] FLAG: --lock-file="" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847024 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847028 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847031 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847037 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847040 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847042 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847045 2576 flags.go:64] FLAG: --logging-format="text" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847048 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847051 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847054 2576 flags.go:64] FLAG: --manifest-url="" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847057 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847061 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847064 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847068 2576 flags.go:64] FLAG: --max-pods="110" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847071 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847074 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847077 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 13:49:52.854018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847080 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847084 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847087 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847090 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847098 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847101 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847104 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847107 2576 flags.go:64] FLAG: --pod-cidr="" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847110 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847117 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847120 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847124 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847127 2576 flags.go:64] FLAG: --port="10250" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847130 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847133 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02e6088a7823708bc" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847136 2576 flags.go:64] FLAG: --qos-reserved="" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847139 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847142 2576 flags.go:64] FLAG: --register-node="true" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847145 2576 flags.go:64] FLAG: --register-with-taints="" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847149 2576 flags.go:64] FLAG: --registry-burst="10" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847152 2576 flags.go:64] FLAG: --registry-qps="5" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847155 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847158 2576 flags.go:64] FLAG: --reserved-memory="" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847162 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847165 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 13:49:52.854644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847168 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847171 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847173 2576 flags.go:64] FLAG: --runonce="false" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847176 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847179 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847182 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847185 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847188 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847191 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847195 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847198 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847201 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847205 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847207 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847210 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847213 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847218 2576 flags.go:64] FLAG: --system-cgroups="" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847221 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847227 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847229 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847232 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847237 2576 flags.go:64] FLAG: --tls-min-version="" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847240 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847243 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847246 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 13:49:52.855501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847249 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847252 2576 flags.go:64] FLAG: --v="2" Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847256 2576 flags.go:64] FLAG: --version="false" Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847260 2576 flags.go:64] FLAG: --vmodule="" Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847264 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.847267 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847401 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847405 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847408 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847411 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847414 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847417 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847420 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847422 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847425 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847429 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847434 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847438 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847442 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847446 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847449 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 13:49:52.856412 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847452 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847455 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847460 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847463 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847466 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847469 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847472 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847475 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847478 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847481 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847484 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847486 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847489 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847491 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847494 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847497 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847499 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847502 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847505 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847507 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 13:49:52.856947 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847510 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847512 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847515 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847518 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847521 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847523 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847526 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847530 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847532 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847535 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847538 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847540 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847543 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847545 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847550 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847554 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847557 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847560 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847563 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847566 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 13:49:52.857438 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847585 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847589 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847592 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847595 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847598 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847600 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847603 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847606 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847609 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847612 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847614 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847617 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847620 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847622 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847625 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847627 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847630 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847632 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847635 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 13:49:52.857949 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847639 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847641 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847644 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847646 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847649 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847652 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847655 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847659 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847662 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847665 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847667 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847670 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847673 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847675 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847678 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847681 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847684 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847686 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847688 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847691 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 13:49:52.858406 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847693 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 13:49:52.858996 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847696 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 13:49:52.858996 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847699 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 13:49:52.858996 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847701 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 13:49:52.858996 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.847703 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 13:49:52.858996 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.848302 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 13:49:52.858996 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.858529 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 13:49:52.858996 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.858554 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 13:49:52.859394 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.859382 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 13:49:52.859426 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.859397 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 13:49:52.859488 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859479 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 13:49:52.859488 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859488 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859491 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859494 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859497 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859501 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859504 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859507 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859510 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859512 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859515 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859518 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859520 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859523 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859525 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859528 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859530 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859533 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859537 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859539 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859543 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 13:49:52.859542 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859547 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859550 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859553 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859557 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859560 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859562 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859565 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859568 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859571 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859574 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859576 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859579 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859583 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859587 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859590 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859593 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859596 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859599 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859602 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 13:49:52.860050 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859605 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859608 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859611 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859613 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859616 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859619 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859623 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859626 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859629 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859632 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859635 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859637 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859640 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859643 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859646 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859649 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859653 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859655 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859658 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859661 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 13:49:52.860503 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859664 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859666 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859669 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859671 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859674 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859677 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859679 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859682 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859685 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859687 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859690 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859693 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859695 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859698 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859700 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859703 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859706 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859708 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859711 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859714 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 13:49:52.861017 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859717 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 13:49:52.861500 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859720 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 13:49:52.861500 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859722 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 13:49:52.861500 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859725 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 13:49:52.861500 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859727 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 13:49:52.861500 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859730 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 13:49:52.861500 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859734 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 13:49:52.861500 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859737 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 13:49:52.861500 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859739 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 13:49:52.861500 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859742 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 13:49:52.861500 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859745 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 13:49:52.861500 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859748 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 13:49:52.861500 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859750 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 13:49:52.861500 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859753 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 13:49:52.861500 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859755 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 13:49:52.861500 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859758 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 13:49:52.861500 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859761 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 13:49:52.861500 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859763 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 13:49:52.861500 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859766 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 13:49:52.861962 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.859771 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 13:49:52.861962 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859922 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 13:49:52.861962 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859928 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 13:49:52.861962 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859932 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 13:49:52.861962 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859935 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 13:49:52.861962 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859937 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 13:49:52.861962 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859940 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 13:49:52.861962 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859943 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 13:49:52.861962 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859946 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 13:49:52.861962 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859949 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 13:49:52.861962 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859952 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 13:49:52.861962 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859954 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 13:49:52.861962 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859957 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 13:49:52.861962 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859960 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 13:49:52.861962 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859963 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859965 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859968 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859970 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859973 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859976 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859979 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859982 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859985 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859988 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859991 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859993 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859996 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.859999 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860001 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860004 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860006 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860009 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860012 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860015 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 13:49:52.862325 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860017 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860020 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860022 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860025 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860027 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860030 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860032 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860035 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860038 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860040 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860043 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860046 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860049 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860051 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860054 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860057 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860060 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860063 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860065 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860068 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 13:49:52.862802 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860070 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860073 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860076 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860078 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860081 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860084 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860086 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860089 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860092 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860096 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860098 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860101 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860103 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860107 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860110 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860113 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860116 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860119 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860121 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 13:49:52.863373 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860125 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860128 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860131 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860134 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860137 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860140 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860143 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860147 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860150 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860152 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860155 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860158 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860160 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860163 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860165 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860168 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860170 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860172 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860175 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860177 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 13:49:52.863859 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860180 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 13:49:52.864331 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860182 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 13:49:52.864331 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860185 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 13:49:52.864331 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860188 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 13:49:52.864331 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860190 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 13:49:52.864331 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860193 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 13:49:52.864331 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:52.860195 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 13:49:52.864331 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.860201 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 13:49:52.864331 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.860358 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 13:49:52.864331 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.862650 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 13:49:52.864331 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.864065 2576 server.go:1019] "Starting client certificate rotation" Jul 10 13:49:52.864331 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.864165 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 13:49:52.864331 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.864208 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 13:49:52.866720 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.866698 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 13:49:52.868114 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.868091 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 13:49:52.879454 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.879435 2576 log.go:25] "Validated CRI v1 runtime API" Jul 10 13:49:52.882431 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.882414 2576 log.go:25] "Validated CRI v1 image API" Jul 10 13:49:52.885783 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.885764 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 13:49:52.888137 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.888117 2576 fs.go:135] Filesystem UUIDs: map[48fbc498-3cff-44c4-ac85-6bbef9c7a8dc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d2587398-cb84-4b78-9a5e-acd600969d6b:/dev/nvme0n1p4] Jul 10 13:49:52.888202 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.888136 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 13:49:52.894436 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.894121 2576 manager.go:216] Machine: {Timestamp:2026-07-10 13:49:52.892639886 +0000 UTC m=+0.356245324 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3083903 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec208f0d0adcfc66c710b94f07381280 SystemUUID:ec208f0d-0adc-fc66-c710-b94f07381280 BootID:1eb08491-b698-402a-80e0-95bee599823b Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f8:95:13:29:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f8:95:13:29:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:45:e0:e4:5e:8c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 13:49:52.894436 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.894422 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 13:49:52.894607 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.894548 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 13:49:52.895877 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.895857 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 13:49:52.897669 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.897639 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 13:49:52.897833 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.897676 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-25.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 13:49:52.898414 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.898402 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 13:49:52.898414 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.898415 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 13:49:52.898485 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.898423 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 13:49:52.899090 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.899079 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 13:49:52.900413 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.900403 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 13:49:52.903878 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.903865 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 10 13:49:52.904017 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.904007 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 13:49:52.904058 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.904038 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 10 13:49:52.904058 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.904047 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 13:49:52.904126 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.904061 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 13:49:52.904126 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.904071 2576 kubelet.go:398] "Adding apiserver pod source" Jul 10 13:49:52.904126 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.904081 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 13:49:52.904489 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.904467 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 13:49:52.904535 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.904491 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 13:49:52.906863 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.906844 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 13:49:52.908897 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.908881 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 13:49:52.908951 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.908913 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 13:49:52.909069 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.909060 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 13:49:52.909099 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.909075 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 13:49:52.909099 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.909081 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 13:49:52.909099 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.909087 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 13:49:52.909099 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.909095 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 13:49:52.909255 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.909101 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 13:49:52.909722 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.909711 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 13:49:52.909722 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.909721 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 13:49:52.909778 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.909728 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 13:49:52.909778 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.909734 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 13:49:52.909778 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.909748 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 13:49:52.909778 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.909757 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 13:49:52.910438 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.910429 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 13:49:52.910438 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.910439 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 13:49:52.915031 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.915016 2576 server.go:1268] "Started kubelet" Jul 10 13:49:52.915196 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.915118 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 13:49:52.915305 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.915219 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 13:49:52.915305 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.915208 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 13:49:52.915994 ip-10-0-143-25 systemd[1]: Started Kubernetes Kubelet. Jul 10 13:49:52.916380 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.916345 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-25.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 13:49:52.916690 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.916673 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 13:49:52.917244 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:52.917218 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-25.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 13:49:52.917311 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:52.917218 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 13:49:52.917566 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.917552 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 10 13:49:52.920722 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:52.920705 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 13:49:52.922589 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.922574 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 13:49:52.922640 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.922579 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 13:49:52.922698 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:52.922680 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-25.ec2.internal\" not found" Jul 10 13:49:52.923194 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:52.922161 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-25.ec2.internal.18c0f1df179e7916 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-25.ec2.internal,UID:ip-10-0-143-25.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-25.ec2.internal,},FirstTimestamp:2026-07-10 13:49:52.914979094 +0000 UTC m=+0.378584530,LastTimestamp:2026-07-10 13:49:52.914979094 +0000 UTC m=+0.378584530,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-25.ec2.internal,}" Jul 10 13:49:52.923340 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.923325 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 13:49:52.923340 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.923325 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 13:49:52.923461 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.923351 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 13:49:52.923461 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.923409 2576 factory.go:55] Registering systemd factory Jul 10 13:49:52.923461 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.923425 2576 factory.go:223] Registration of the systemd container factory successfully Jul 10 13:49:52.923591 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.923471 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 10 13:49:52.923591 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.923482 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 10 13:49:52.923669 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.923659 2576 factory.go:153] Registering CRI-O factory Jul 10 13:49:52.923713 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.923671 2576 factory.go:223] Registration of the crio container factory successfully Jul 10 13:49:52.923756 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.923724 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 13:49:52.923756 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.923748 2576 factory.go:103] Registering Raw factory Jul 10 13:49:52.923839 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.923763 2576 manager.go:1196] Started watching for new ooms in manager Jul 10 13:49:52.924828 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.924797 2576 manager.go:349] Starting recovery of all containers Jul 10 13:49:52.931485 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:52.931440 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-25.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 13:49:52.931710 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:52.931686 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 13:49:52.934432 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.934248 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 13:49:52.937470 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.937454 2576 manager.go:354] Recovery completed Jul 10 13:49:52.942380 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.942367 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:49:52.944298 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.944280 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:49:52.944368 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.944312 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:49:52.944368 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.944323 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:49:52.944926 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.944913 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 13:49:52.944926 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.944920 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:49:52.945036 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.944923 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 13:49:52.945036 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.944996 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 10 13:49:52.945662 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.945648 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:49:52.945730 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.945678 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:49:52.945730 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.945693 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:49:52.946341 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.946328 2576 policy_none.go:49] "None policy: Start" Jul 10 13:49:52.946385 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.946345 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 13:49:52.946385 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.946355 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 13:49:52.946737 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:52.946672 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-25.ec2.internal.18c0f1df195dde65 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-25.ec2.internal,UID:ip-10-0-143-25.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-25.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-25.ec2.internal,},FirstTimestamp:2026-07-10 13:49:52.944299621 +0000 UTC m=+0.407905060,LastTimestamp:2026-07-10 13:49:52.944299621 +0000 UTC m=+0.407905060,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-25.ec2.internal,}" Jul 10 13:49:52.947865 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.947855 2576 policy_none.go:47] "Start" Jul 10 13:49:52.963074 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:52.962981 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-25.ec2.internal.18c0f1df195e2476 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-25.ec2.internal,UID:ip-10-0-143-25.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-25.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-25.ec2.internal,},FirstTimestamp:2026-07-10 13:49:52.944317558 +0000 UTC m=+0.407922994,LastTimestamp:2026-07-10 13:49:52.944317558 +0000 UTC m=+0.407922994,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-25.ec2.internal,}" Jul 10 13:49:52.963578 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.963559 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vchn8" Jul 10 13:49:52.974536 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:52.974461 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-25.ec2.internal.18c0f1df195e49af default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-25.ec2.internal,UID:ip-10-0-143-25.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-25.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-25.ec2.internal,},FirstTimestamp:2026-07-10 13:49:52.944327087 +0000 UTC m=+0.407932523,LastTimestamp:2026-07-10 13:49:52.944327087 +0000 UTC m=+0.407932523,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-25.ec2.internal,}" Jul 10 13:49:52.975322 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.975300 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vchn8" Jul 10 13:49:52.986580 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.986564 2576 manager.go:337] "Starting Device Plugin manager" Jul 10 13:49:53.004887 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:52.986613 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 13:49:53.004887 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.986627 2576 server.go:87] "Starting device plugin registration server" Jul 10 13:49:53.004887 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.986925 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 13:49:53.004887 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.986935 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 13:49:53.004887 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.987047 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 13:49:53.004887 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.987119 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 13:49:53.004887 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:52.987128 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 13:49:53.004887 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:52.987684 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 13:49:53.004887 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:52.987735 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-25.ec2.internal\" not found" Jul 10 13:49:53.029754 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.029720 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 13:49:53.029934 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.029768 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 13:49:53.029934 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.029790 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 13:49:53.029934 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:53.029838 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 13:49:53.032494 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.032470 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 13:49:53.087865 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.087824 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:49:53.089061 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.089011 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:49:53.089061 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.089042 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:49:53.089061 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.089052 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:49:53.089197 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.089095 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-25.ec2.internal" Jul 10 13:49:53.096597 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.096577 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-25.ec2.internal" Jul 10 13:49:53.096654 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:53.096605 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-25.ec2.internal\": node \"ip-10-0-143-25.ec2.internal\" not found" Jul 10 13:49:53.114782 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:53.114748 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-25.ec2.internal\" not found" Jul 10 13:49:53.130597 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.130566 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-25.ec2.internal"] Jul 10 13:49:53.130662 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.130643 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:49:53.132240 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.132224 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:49:53.132311 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.132253 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:49:53.132311 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.132263 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:49:53.133494 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.133483 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:49:53.133632 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.133616 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal" Jul 10 13:49:53.133665 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.133656 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:49:53.134272 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.134257 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:49:53.134342 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.134283 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:49:53.134342 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.134297 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:49:53.134342 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.134285 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:49:53.134430 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.134360 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:49:53.134430 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.134373 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:49:53.135571 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.135556 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-25.ec2.internal" Jul 10 13:49:53.135623 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.135583 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:49:53.136237 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.136222 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:49:53.136342 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.136251 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:49:53.136342 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.136263 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:49:53.162683 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:53.162658 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-25.ec2.internal\" not found" node="ip-10-0-143-25.ec2.internal" Jul 10 13:49:53.168684 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:53.168657 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-25.ec2.internal\" not found" node="ip-10-0-143-25.ec2.internal" Jul 10 13:49:53.215826 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:53.215782 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-25.ec2.internal\" not found" Jul 10 13:49:53.225535 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.225511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a343a858d6edb11abb86474c850c3d4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal\" (UID: \"a343a858d6edb11abb86474c850c3d4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal" Jul 10 13:49:53.225625 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.225541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a343a858d6edb11abb86474c850c3d4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal\" (UID: \"a343a858d6edb11abb86474c850c3d4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal" Jul 10 13:49:53.225625 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.225564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b15c5c2738510537812bfa649878a461-config\") pod \"kube-apiserver-proxy-ip-10-0-143-25.ec2.internal\" (UID: \"b15c5c2738510537812bfa649878a461\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-25.ec2.internal" Jul 10 13:49:53.316692 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:53.316640 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-25.ec2.internal\" not found" Jul 10 13:49:53.326032 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.326003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a343a858d6edb11abb86474c850c3d4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal\" (UID: \"a343a858d6edb11abb86474c850c3d4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal" Jul 10 13:49:53.326106 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.326039 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a343a858d6edb11abb86474c850c3d4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal\" (UID: \"a343a858d6edb11abb86474c850c3d4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal" Jul 10 13:49:53.326106 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.326062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b15c5c2738510537812bfa649878a461-config\") pod \"kube-apiserver-proxy-ip-10-0-143-25.ec2.internal\" (UID: \"b15c5c2738510537812bfa649878a461\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-25.ec2.internal" Jul 10 13:49:53.326203 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.326127 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b15c5c2738510537812bfa649878a461-config\") pod \"kube-apiserver-proxy-ip-10-0-143-25.ec2.internal\" (UID: \"b15c5c2738510537812bfa649878a461\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-25.ec2.internal" Jul 10 13:49:53.326203 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.326153 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a343a858d6edb11abb86474c850c3d4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal\" (UID: \"a343a858d6edb11abb86474c850c3d4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal" Jul 10 13:49:53.326203 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.326125 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a343a858d6edb11abb86474c850c3d4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal\" (UID: \"a343a858d6edb11abb86474c850c3d4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal" Jul 10 13:49:53.417481 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:53.417450 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-25.ec2.internal\" not found" Jul 10 13:49:53.464959 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.464933 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal" Jul 10 13:49:53.471954 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.471936 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-25.ec2.internal" Jul 10 13:49:53.518488 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:53.518453 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-25.ec2.internal\" not found" Jul 10 13:49:53.618940 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:53.618917 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-25.ec2.internal\" not found" Jul 10 13:49:53.719577 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:53.719505 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-25.ec2.internal\" not found" Jul 10 13:49:53.748155 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.748128 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 13:49:53.820498 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:53.820468 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-25.ec2.internal\" not found" Jul 10 13:49:53.863950 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.863919 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 13:49:53.864574 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.864110 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 13:49:53.864574 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.864135 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 13:49:53.921509 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:53.921481 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-25.ec2.internal\" not found" Jul 10 13:49:53.932401 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.932374 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 13:49:53.978305 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.978226 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 13:44:53 +0000 UTC" deadline="2028-03-16 08:26:07.448951065 +0000 UTC" Jul 10 13:49:53.978305 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:53.978261 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14754h36m13.47069378s" Jul 10 13:49:54.022566 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:54.022535 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-25.ec2.internal\" not found" Jul 10 13:49:54.061753 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.061731 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:49:54.063415 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.063397 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:49:54.063510 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.063427 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:49:54.063510 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.063438 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:49:54.066093 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:54.066069 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda343a858d6edb11abb86474c850c3d4f.slice/crio-d8051baef83018c204b031890ed301353c25d8efe3b3f15fa0ee6ab5d6cec9af WatchSource:0}: Error finding container d8051baef83018c204b031890ed301353c25d8efe3b3f15fa0ee6ab5d6cec9af: Status 404 returned error can't find the container with id d8051baef83018c204b031890ed301353c25d8efe3b3f15fa0ee6ab5d6cec9af Jul 10 13:49:54.066355 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:54.066336 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15c5c2738510537812bfa649878a461.slice/crio-238959b93ac16c038ca0b8169110c55d13830aa46d7109ea8e09e64f50a2a885 WatchSource:0}: Error finding container 238959b93ac16c038ca0b8169110c55d13830aa46d7109ea8e09e64f50a2a885: Status 404 returned error can't find the container with id 238959b93ac16c038ca0b8169110c55d13830aa46d7109ea8e09e64f50a2a885 Jul 10 13:49:54.070528 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.070513 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 13:49:54.123204 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:54.123166 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-25.ec2.internal\" not found" Jul 10 13:49:54.223654 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:54.223624 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-25.ec2.internal\" not found" Jul 10 13:49:54.324181 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:54.324094 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-25.ec2.internal\" not found" Jul 10 13:49:54.340238 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.340213 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 13:49:54.422762 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.422721 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal" Jul 10 13:49:54.442174 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.442145 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 13:49:54.443366 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.443351 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-25.ec2.internal" Jul 10 13:49:54.452686 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.452662 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 13:49:54.741179 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.741150 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 13:49:54.906319 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.906284 2576 apiserver.go:52] "Watching apiserver" Jul 10 13:49:54.919502 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.919468 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 13:49:54.920006 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.919976 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-4qc7h","kube-system/global-pull-secret-syncer-zbx2d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd","openshift-image-registry/node-ca-9nsfh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal","openshift-multus/network-metrics-daemon-zpqdz","openshift-ovn-kubernetes/ovnkube-node-r5xc7","kube-system/konnectivity-agent-5wgd6","kube-system/kube-apiserver-proxy-ip-10-0-143-25.ec2.internal","openshift-cluster-node-tuning-operator/tuned-6pk5v","openshift-dns/node-resolver-v6qp5","openshift-multus/multus-4v5nf","openshift-multus/multus-additional-cni-plugins-jmkqc","openshift-network-diagnostics/network-check-target-hv628"] Jul 10 13:49:54.922461 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.921997 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.923934 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.923520 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v6qp5" Jul 10 13:49:54.923934 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.923781 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 13:49:54.926352 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.926074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:54.927823 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.927794 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9nsfh" Jul 10 13:49:54.928746 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.928719 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rhrct\"" Jul 10 13:49:54.928857 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.928787 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 13:49:54.929150 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.929079 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:49:54.929258 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:54.929168 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv628" podUID="f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00" Jul 10 13:49:54.931342 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.931314 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 13:49:54.931600 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.931587 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:49:54.931687 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:54.931662 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpqdz" podUID="572667f8-51ec-40b5-8678-d1373066ffac" Jul 10 13:49:54.932475 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.932431 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 13:49:54.933293 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.933266 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5wgd6" Jul 10 13:49:54.934050 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.933855 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 13:49:54.934050 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.933861 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 13:49:54.934050 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.933915 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2qjlb\"" Jul 10 13:49:54.934050 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.934000 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 13:49:54.934050 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.934046 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 13:49:54.934480 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.934234 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 13:49:54.934480 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.934279 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 13:49:54.934480 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.934420 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 13:49:54.935290 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.935264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-run-systemd\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.935828 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.935800 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-ovnkube-config\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.935957 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.935863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-sys-fs\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:54.935957 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.935947 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-var-lib-openvswitch\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.936065 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.935984 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-run-ovn-kubernetes\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.936065 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.936019 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-device-dir\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:54.936163 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.936071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-systemd-units\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.936163 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.936104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.936287 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.936258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5st5s\" (UniqueName: \"kubernetes.io/projected/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-kube-api-access-5st5s\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.936338 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.936290 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-socket-dir\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:54.936338 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.936320 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-node-log\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.936575 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.936536 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 13:49:54.937461 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.937438 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gdkzq\"" Jul 10 13:49:54.938039 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.937682 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 13:49:54.938383 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.936349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-cni-netd\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.938503 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.938424 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-ovnkube-script-lib\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.938503 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.938461 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3c3e7158-27d5-4642-b99f-7f2a8a728726-hosts-file\") pod \"node-resolver-v6qp5\" (UID: \"3c3e7158-27d5-4642-b99f-7f2a8a728726\") " pod="openshift-dns/node-resolver-v6qp5" Jul 10 13:49:54.938580 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.938490 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tv7j\" (UniqueName: \"kubernetes.io/projected/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-kube-api-access-2tv7j\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:54.938580 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.938571 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-log-socket\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.938644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.938608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-run-openvswitch\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.938677 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.938639 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-run-ovn\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.938677 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.938667 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-ovn-node-metrics-cert\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.938735 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.938698 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3e7158-27d5-4642-b99f-7f2a8a728726-tmp-dir\") pod \"node-resolver-v6qp5\" (UID: \"3c3e7158-27d5-4642-b99f-7f2a8a728726\") " pod="openshift-dns/node-resolver-v6qp5" Jul 10 13:49:54.938856 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.938733 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnmn7\" (UniqueName: \"kubernetes.io/projected/3c3e7158-27d5-4642-b99f-7f2a8a728726-kube-api-access-gnmn7\") pod \"node-resolver-v6qp5\" (UID: \"3c3e7158-27d5-4642-b99f-7f2a8a728726\") " pod="openshift-dns/node-resolver-v6qp5" Jul 10 13:49:54.938964 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.938877 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:54.938964 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.938910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-registration-dir\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:54.938964 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.938934 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-kubelet\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.939112 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.938967 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-slash\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.939112 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.939026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-env-overrides\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.939112 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.939054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-run-netns\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.939305 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.939123 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 13:49:54.939305 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.939146 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-etc-openvswitch\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.939305 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.939172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-cni-bin\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:54.939305 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.939210 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-etc-selinux\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:54.939305 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.939271 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 13:49:54.940978 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.940717 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m4j4j\"" Jul 10 13:49:54.940978 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.940973 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h6w6s\"" Jul 10 13:49:54.941160 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.941126 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:49:54.941237 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.941197 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 13:49:54.941237 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:54.941206 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbx2d" podUID="c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4" Jul 10 13:49:54.941237 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.941211 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:54.941399 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.941287 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 13:49:54.942407 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.942378 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:54.944460 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.944387 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4qc7h" Jul 10 13:49:54.945095 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.944910 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9q5wf\"" Jul 10 13:49:54.945309 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.945290 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 13:49:54.945423 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.945400 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 13:49:54.945526 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.945503 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 13:49:54.946622 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.945771 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4v5nf" Jul 10 13:49:54.946622 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.946282 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 13:49:54.946622 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.946305 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 13:49:54.947061 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.946998 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 13:49:54.947061 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.947029 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w2wkd\"" Jul 10 13:49:54.947061 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.947037 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 13:49:54.947288 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.947273 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 13:49:54.948626 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.948158 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 13:49:54.948626 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.948421 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 13:49:54.948626 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.948483 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 13:49:54.948626 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.948530 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 13:49:54.948626 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.948485 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5h7t4\"" Jul 10 13:49:54.948967 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.948747 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bw95n\"" Jul 10 13:49:54.994074 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:54.994005 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2czn6" Jul 10 13:49:55.012721 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.012686 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2czn6" Jul 10 13:49:55.025534 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.025509 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 13:49:55.038660 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.038593 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal" event={"ID":"a343a858d6edb11abb86474c850c3d4f","Type":"ContainerStarted","Data":"d8051baef83018c204b031890ed301353c25d8efe3b3f15fa0ee6ab5d6cec9af"} Jul 10 13:49:55.039410 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039379 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-cni-netd\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.039549 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tv7j\" (UniqueName: \"kubernetes.io/projected/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-kube-api-access-2tv7j\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:55.039549 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039454 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjtdm\" (UniqueName: \"kubernetes.io/projected/868d4bcd-6eec-475b-a138-12fa9712ae87-kube-api-access-qjtdm\") pod \"node-ca-9nsfh\" (UID: \"868d4bcd-6eec-475b-a138-12fa9712ae87\") " pod="openshift-image-registry/node-ca-9nsfh" Jul 10 13:49:55.039549 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039490 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-cni-binary-copy\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.039549 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039497 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-cni-netd\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.039549 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.039549 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-kubelet-config\") pod \"global-pull-secret-syncer-zbx2d\" (UID: \"c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4\") " pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:49:55.040004 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039573 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-host-var-lib-kubelet\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.040004 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039599 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-ovn-node-metrics-cert\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.040004 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039625 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnmn7\" (UniqueName: \"kubernetes.io/projected/3c3e7158-27d5-4642-b99f-7f2a8a728726-kube-api-access-gnmn7\") pod \"node-resolver-v6qp5\" (UID: \"3c3e7158-27d5-4642-b99f-7f2a8a728726\") " pod="openshift-dns/node-resolver-v6qp5" Jul 10 13:49:55.040004 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-system-cni-dir\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.040004 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039676 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aea6d43a-5030-4d5b-89e7-c8dfd95b0b27-agent-certs\") pod \"konnectivity-agent-5wgd6\" (UID: \"aea6d43a-5030-4d5b-89e7-c8dfd95b0b27\") " pod="kube-system/konnectivity-agent-5wgd6" Jul 10 13:49:55.040004 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51d53d7f-8689-454d-a7e4-b6eec9c9c756-multus-daemon-config\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.040004 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039728 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:55.040004 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039760 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl2ff\" (UniqueName: \"kubernetes.io/projected/572667f8-51ec-40b5-8678-d1373066ffac-kube-api-access-rl2ff\") pod \"network-metrics-daemon-zpqdz\" (UID: \"572667f8-51ec-40b5-8678-d1373066ffac\") " pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:49:55.040004 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-kubelet\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.040004 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039815 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-env-overrides\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.040004 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039847 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-sysconfig\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.040004 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039889 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-lib-modules\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.040004 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn42f\" (UniqueName: \"kubernetes.io/projected/4e5cb550-3f73-42da-b737-5ebaae37e370-kube-api-access-hn42f\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.040004 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039938 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-hostroot\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.040004 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039965 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-run-netns\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.040004 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039987 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:55.040004 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.039991 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-etc-openvswitch\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.041030 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040040 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-etc-openvswitch\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.041030 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040048 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-etc-selinux\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:55.041030 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040086 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.041030 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040085 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 13:49:55.041030 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-dbus\") pod \"global-pull-secret-syncer-zbx2d\" (UID: \"c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4\") " pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:49:55.041030 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret\") pod \"global-pull-secret-syncer-zbx2d\" (UID: \"c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4\") " pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:49:55.041030 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040177 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-sys\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.041030 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040204 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-var-lib-kubelet\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.041030 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040231 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-run-systemd\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.041030 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040255 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/868d4bcd-6eec-475b-a138-12fa9712ae87-host\") pod \"node-ca-9nsfh\" (UID: \"868d4bcd-6eec-475b-a138-12fa9712ae87\") " pod="openshift-image-registry/node-ca-9nsfh" Jul 10 13:49:55.041030 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040281 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs\") pod \"network-metrics-daemon-zpqdz\" (UID: \"572667f8-51ec-40b5-8678-d1373066ffac\") " pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:49:55.041030 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040380 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-run-systemd\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.041030 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040588 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-env-overrides\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.041030 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-etc-selinux\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:55.041030 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040705 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-run-netns\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.041030 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040712 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4103f91e-397c-4638-9754-b5d9567f1f22-host-slash\") pod \"iptables-alerter-4qc7h\" (UID: \"4103f91e-397c-4638-9754-b5d9567f1f22\") " pod="openshift-network-operator/iptables-alerter-4qc7h" Jul 10 13:49:55.041030 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040737 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-multus-cni-dir\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.041941 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-os-release\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.041941 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040785 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-multus-conf-dir\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.041941 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sb28\" (UniqueName: \"kubernetes.io/projected/51d53d7f-8689-454d-a7e4-b6eec9c9c756-kube-api-access-4sb28\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.041941 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040110 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-kubelet\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.041941 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-var-lib-openvswitch\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.041941 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040833 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-25.ec2.internal" event={"ID":"b15c5c2738510537812bfa649878a461","Type":"ContainerStarted","Data":"238959b93ac16c038ca0b8169110c55d13830aa46d7109ea8e09e64f50a2a885"} Jul 10 13:49:55.041941 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040939 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-var-lib-openvswitch\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.041941 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-run-ovn-kubernetes\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.041941 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.040978 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-run-ovn-kubernetes\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.041941 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.041384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/868d4bcd-6eec-475b-a138-12fa9712ae87-serviceca\") pod \"node-ca-9nsfh\" (UID: \"868d4bcd-6eec-475b-a138-12fa9712ae87\") " pod="openshift-image-registry/node-ca-9nsfh" Jul 10 13:49:55.041941 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.041432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-tuned\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.041941 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.041463 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-multus-socket-dir-parent\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.041941 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.041492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-host-var-lib-cni-multus\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.041941 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.041522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-systemd-units\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.041941 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.041601 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5st5s\" (UniqueName: \"kubernetes.io/projected/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-kube-api-access-5st5s\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.041941 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.041629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-socket-dir\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:55.041941 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.041655 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-cnibin\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.042737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.041681 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-modprobe-d\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.042737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.041706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-systemd\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.042737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.041632 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-systemd-units\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.042737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.041733 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-host-run-multus-certs\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.042737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.041771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-node-log\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.042737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.041835 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-node-log\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.042737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.041851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-ovnkube-script-lib\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.042737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.041886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3c3e7158-27d5-4642-b99f-7f2a8a728726-hosts-file\") pod \"node-resolver-v6qp5\" (UID: \"3c3e7158-27d5-4642-b99f-7f2a8a728726\") " pod="openshift-dns/node-resolver-v6qp5" Jul 10 13:49:55.042737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.041932 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-sysctl-d\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.042737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.041976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51d53d7f-8689-454d-a7e4-b6eec9c9c756-cni-binary-copy\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.042737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-log-socket\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.042737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042004 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-socket-dir\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:55.042737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-run-openvswitch\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.042737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042075 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-log-socket\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.042737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-run-ovn\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.042737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042114 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3c3e7158-27d5-4642-b99f-7f2a8a728726-hosts-file\") pod \"node-resolver-v6qp5\" (UID: \"3c3e7158-27d5-4642-b99f-7f2a8a728726\") " pod="openshift-dns/node-resolver-v6qp5" Jul 10 13:49:55.042737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042118 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-run-openvswitch\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.042737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042151 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-run-ovn\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.043564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3e7158-27d5-4642-b99f-7f2a8a728726-tmp-dir\") pod \"node-resolver-v6qp5\" (UID: \"3c3e7158-27d5-4642-b99f-7f2a8a728726\") " pod="openshift-dns/node-resolver-v6qp5" Jul 10 13:49:55.043564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042201 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-kubernetes\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.043564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-system-cni-dir\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.043564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042275 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-host-run-netns\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.043564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-registration-dir\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:55.043564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042340 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wdpv\" (UniqueName: \"kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv\") pod \"network-check-target-hv628\" (UID: \"f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00\") " pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:49:55.043564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042369 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-sysctl-conf\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.043564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042382 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-ovnkube-script-lib\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.043564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042454 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3e7158-27d5-4642-b99f-7f2a8a728726-tmp-dir\") pod \"node-resolver-v6qp5\" (UID: \"3c3e7158-27d5-4642-b99f-7f2a8a728726\") " pod="openshift-dns/node-resolver-v6qp5" Jul 10 13:49:55.043564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-etc-kubernetes\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.043564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042521 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-registration-dir\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:55.043564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042576 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-slash\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.043564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042601 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-os-release\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.043564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042643 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-slash\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.043564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042657 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e5cb550-3f73-42da-b737-5ebaae37e370-tmp\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.043564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042694 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwdnw\" (UniqueName: \"kubernetes.io/projected/4103f91e-397c-4638-9754-b5d9567f1f22-kube-api-access-mwdnw\") pod \"iptables-alerter-4qc7h\" (UID: \"4103f91e-397c-4638-9754-b5d9567f1f22\") " pod="openshift-network-operator/iptables-alerter-4qc7h" Jul 10 13:49:55.043564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-cni-bin\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.044490 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042755 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.044490 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042787 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-cnibin\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.044490 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042852 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-cni-bin\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.044490 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042887 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-ovnkube-config\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.044490 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-sys-fs\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:55.044490 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.042998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-device-dir\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:55.044490 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.043047 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55rrc\" (UniqueName: \"kubernetes.io/projected/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-kube-api-access-55rrc\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.044490 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.043078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-run\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.044490 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.043102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-host\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.044490 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.043181 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-sys-fs\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:55.044490 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.043201 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-device-dir\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:55.044490 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.043202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4103f91e-397c-4638-9754-b5d9567f1f22-iptables-alerter-script\") pod \"iptables-alerter-4qc7h\" (UID: \"4103f91e-397c-4638-9754-b5d9567f1f22\") " pod="openshift-network-operator/iptables-alerter-4qc7h" Jul 10 13:49:55.044490 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.043253 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-host-run-k8s-cni-cncf-io\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.044490 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.043286 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.044490 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.043326 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aea6d43a-5030-4d5b-89e7-c8dfd95b0b27-konnectivity-ca\") pod \"konnectivity-agent-5wgd6\" (UID: \"aea6d43a-5030-4d5b-89e7-c8dfd95b0b27\") " pod="kube-system/konnectivity-agent-5wgd6" Jul 10 13:49:55.044490 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.043352 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.045556 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.043353 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-host-var-lib-cni-bin\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.045556 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.043421 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-ovnkube-config\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.045556 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.044288 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-ovn-node-metrics-cert\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.052730 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.052613 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tv7j\" (UniqueName: \"kubernetes.io/projected/109c0a7d-50e6-4d3d-ac0a-fb27c253bcba-kube-api-access-2tv7j\") pod \"aws-ebs-csi-driver-node-nlrdd\" (UID: \"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:55.052730 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.052622 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5st5s\" (UniqueName: \"kubernetes.io/projected/2a5944e2-4460-42a5-9e6e-4bbbc51f60ed-kube-api-access-5st5s\") pod \"ovnkube-node-r5xc7\" (UID: \"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.054119 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.054091 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnmn7\" (UniqueName: \"kubernetes.io/projected/3c3e7158-27d5-4642-b99f-7f2a8a728726-kube-api-access-gnmn7\") pod \"node-resolver-v6qp5\" (UID: \"3c3e7158-27d5-4642-b99f-7f2a8a728726\") " pod="openshift-dns/node-resolver-v6qp5" Jul 10 13:49:55.143826 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.143739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/868d4bcd-6eec-475b-a138-12fa9712ae87-host\") pod \"node-ca-9nsfh\" (UID: \"868d4bcd-6eec-475b-a138-12fa9712ae87\") " pod="openshift-image-registry/node-ca-9nsfh" Jul 10 13:49:55.143826 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.143782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs\") pod \"network-metrics-daemon-zpqdz\" (UID: \"572667f8-51ec-40b5-8678-d1373066ffac\") " pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:49:55.143826 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.143813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4103f91e-397c-4638-9754-b5d9567f1f22-host-slash\") pod \"iptables-alerter-4qc7h\" (UID: \"4103f91e-397c-4638-9754-b5d9567f1f22\") " pod="openshift-network-operator/iptables-alerter-4qc7h" Jul 10 13:49:55.143826 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.143809 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/868d4bcd-6eec-475b-a138-12fa9712ae87-host\") pod \"node-ca-9nsfh\" (UID: \"868d4bcd-6eec-475b-a138-12fa9712ae87\") " pod="openshift-image-registry/node-ca-9nsfh" Jul 10 13:49:55.144179 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.143852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-multus-cni-dir\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.144179 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.143880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-os-release\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.144179 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.143901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-multus-conf-dir\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.144179 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.143924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sb28\" (UniqueName: \"kubernetes.io/projected/51d53d7f-8689-454d-a7e4-b6eec9c9c756-kube-api-access-4sb28\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.144179 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.143952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/868d4bcd-6eec-475b-a138-12fa9712ae87-serviceca\") pod \"node-ca-9nsfh\" (UID: \"868d4bcd-6eec-475b-a138-12fa9712ae87\") " pod="openshift-image-registry/node-ca-9nsfh" Jul 10 13:49:55.144179 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:55.143955 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:49:55.144179 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.143975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-tuned\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.144179 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144002 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-multus-socket-dir-parent\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.144179 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:55.144056 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs podName:572667f8-51ec-40b5-8678-d1373066ffac nodeName:}" failed. No retries permitted until 2026-07-10 13:49:55.644013083 +0000 UTC m=+3.107618537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs") pod "network-metrics-daemon-zpqdz" (UID: "572667f8-51ec-40b5-8678-d1373066ffac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:49:55.144179 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144063 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-multus-conf-dir\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.144179 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144078 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-host-var-lib-cni-multus\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.144179 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144115 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-cnibin\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.144179 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144133 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-multus-socket-dir-parent\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.144179 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-modprobe-d\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.144179 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-systemd\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.144949 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-host-run-multus-certs\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.144949 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144219 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-sysctl-d\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.144949 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144242 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51d53d7f-8689-454d-a7e4-b6eec9c9c756-cni-binary-copy\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.144949 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144263 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-modprobe-d\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.144949 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144274 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-kubernetes\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.144949 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-system-cni-dir\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.144949 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144319 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-host-run-netns\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.144949 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wdpv\" (UniqueName: \"kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv\") pod \"network-check-target-hv628\" (UID: \"f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00\") " pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:49:55.144949 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144369 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-sysctl-conf\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.144949 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144393 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-etc-kubernetes\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.144949 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-os-release\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.144949 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e5cb550-3f73-42da-b737-5ebaae37e370-tmp\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.144949 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144452 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-cnibin\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.144949 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144471 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwdnw\" (UniqueName: \"kubernetes.io/projected/4103f91e-397c-4638-9754-b5d9567f1f22-kube-api-access-mwdnw\") pod \"iptables-alerter-4qc7h\" (UID: \"4103f91e-397c-4638-9754-b5d9567f1f22\") " pod="openshift-network-operator/iptables-alerter-4qc7h" Jul 10 13:49:55.144949 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144510 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-systemd\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.144949 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144516 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.144949 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144528 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-kubernetes\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.145755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144454 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4103f91e-397c-4638-9754-b5d9567f1f22-host-slash\") pod \"iptables-alerter-4qc7h\" (UID: \"4103f91e-397c-4638-9754-b5d9567f1f22\") " pod="openshift-network-operator/iptables-alerter-4qc7h" Jul 10 13:49:55.145755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144576 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-host-var-lib-cni-multus\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.145755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144589 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-multus-cni-dir\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.145755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144630 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-host-run-multus-certs\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.145755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144672 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-system-cni-dir\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.145755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144676 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-os-release\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.145755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144796 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-host-run-netns\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.145755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144806 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-sysctl-d\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.145755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144823 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-sysctl-conf\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.145755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144832 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/868d4bcd-6eec-475b-a138-12fa9712ae87-serviceca\") pod \"node-ca-9nsfh\" (UID: \"868d4bcd-6eec-475b-a138-12fa9712ae87\") " pod="openshift-image-registry/node-ca-9nsfh" Jul 10 13:49:55.145755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144912 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-etc-kubernetes\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.145755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-cnibin\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.145755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144989 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.145755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.144986 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55rrc\" (UniqueName: \"kubernetes.io/projected/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-kube-api-access-55rrc\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.145755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-run\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.145755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145047 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-cnibin\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.145755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-host\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.145755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145083 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4103f91e-397c-4638-9754-b5d9567f1f22-iptables-alerter-script\") pod \"iptables-alerter-4qc7h\" (UID: \"4103f91e-397c-4638-9754-b5d9567f1f22\") " pod="openshift-network-operator/iptables-alerter-4qc7h" Jul 10 13:49:55.146673 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-host-run-k8s-cni-cncf-io\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.146673 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aea6d43a-5030-4d5b-89e7-c8dfd95b0b27-konnectivity-ca\") pod \"konnectivity-agent-5wgd6\" (UID: \"aea6d43a-5030-4d5b-89e7-c8dfd95b0b27\") " pod="kube-system/konnectivity-agent-5wgd6" Jul 10 13:49:55.146673 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-host-var-lib-cni-bin\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.146673 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjtdm\" (UniqueName: \"kubernetes.io/projected/868d4bcd-6eec-475b-a138-12fa9712ae87-kube-api-access-qjtdm\") pod \"node-ca-9nsfh\" (UID: \"868d4bcd-6eec-475b-a138-12fa9712ae87\") " pod="openshift-image-registry/node-ca-9nsfh" Jul 10 13:49:55.146673 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-cni-binary-copy\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.146673 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145248 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.146673 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-kubelet-config\") pod \"global-pull-secret-syncer-zbx2d\" (UID: \"c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4\") " pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:49:55.146673 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-host-var-lib-kubelet\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.146673 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145334 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-host-run-k8s-cni-cncf-io\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.146673 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145351 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51d53d7f-8689-454d-a7e4-b6eec9c9c756-cni-binary-copy\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.146673 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-system-cni-dir\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.146673 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aea6d43a-5030-4d5b-89e7-c8dfd95b0b27-agent-certs\") pod \"konnectivity-agent-5wgd6\" (UID: \"aea6d43a-5030-4d5b-89e7-c8dfd95b0b27\") " pod="kube-system/konnectivity-agent-5wgd6" Jul 10 13:49:55.146673 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145408 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51d53d7f-8689-454d-a7e4-b6eec9c9c756-multus-daemon-config\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.146673 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145439 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl2ff\" (UniqueName: \"kubernetes.io/projected/572667f8-51ec-40b5-8678-d1373066ffac-kube-api-access-rl2ff\") pod \"network-metrics-daemon-zpqdz\" (UID: \"572667f8-51ec-40b5-8678-d1373066ffac\") " pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:49:55.146673 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145467 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-sysconfig\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.146673 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145469 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-kubelet-config\") pod \"global-pull-secret-syncer-zbx2d\" (UID: \"c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4\") " pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:49:55.146673 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145494 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-lib-modules\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.147470 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145521 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn42f\" (UniqueName: \"kubernetes.io/projected/4e5cb550-3f73-42da-b737-5ebaae37e370-kube-api-access-hn42f\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.147470 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145522 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-host-var-lib-cni-bin\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.147470 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145594 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-system-cni-dir\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.147470 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-hostroot\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.147470 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145792 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4103f91e-397c-4638-9754-b5d9567f1f22-iptables-alerter-script\") pod \"iptables-alerter-4qc7h\" (UID: \"4103f91e-397c-4638-9754-b5d9567f1f22\") " pod="openshift-network-operator/iptables-alerter-4qc7h" Jul 10 13:49:55.147470 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145793 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.147470 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145859 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-dbus\") pod \"global-pull-secret-syncer-zbx2d\" (UID: \"c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4\") " pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:49:55.147470 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret\") pod \"global-pull-secret-syncer-zbx2d\" (UID: \"c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4\") " pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:49:55.147470 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-sys\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.147470 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-var-lib-kubelet\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.147470 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145974 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.147470 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.146037 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aea6d43a-5030-4d5b-89e7-c8dfd95b0b27-konnectivity-ca\") pod \"konnectivity-agent-5wgd6\" (UID: \"aea6d43a-5030-4d5b-89e7-c8dfd95b0b27\") " pod="kube-system/konnectivity-agent-5wgd6" Jul 10 13:49:55.147470 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.146068 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-var-lib-kubelet\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.147470 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:55.146087 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:49:55.147470 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.146112 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-host\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.147470 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:55.146138 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret podName:c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4 nodeName:}" failed. No retries permitted until 2026-07-10 13:49:55.646122362 +0000 UTC m=+3.109727805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret") pod "global-pull-secret-syncer-zbx2d" (UID: "c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:49:55.147470 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.146161 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-sys\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.148279 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.146224 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-os-release\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.148279 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.146261 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-dbus\") pod \"global-pull-secret-syncer-zbx2d\" (UID: \"c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4\") " pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:49:55.148279 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.146301 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.148279 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.145404 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-run\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.148279 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.146368 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-hostroot\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.148279 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.146385 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51d53d7f-8689-454d-a7e4-b6eec9c9c756-host-var-lib-kubelet\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.148279 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.146528 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-lib-modules\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.148279 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.146583 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-sysconfig\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.148279 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.146707 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-cni-binary-copy\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.148279 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.146831 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51d53d7f-8689-454d-a7e4-b6eec9c9c756-multus-daemon-config\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.148279 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.147792 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e5cb550-3f73-42da-b737-5ebaae37e370-etc-tuned\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.148683 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.148369 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e5cb550-3f73-42da-b737-5ebaae37e370-tmp\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.149163 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.149139 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aea6d43a-5030-4d5b-89e7-c8dfd95b0b27-agent-certs\") pod \"konnectivity-agent-5wgd6\" (UID: \"aea6d43a-5030-4d5b-89e7-c8dfd95b0b27\") " pod="kube-system/konnectivity-agent-5wgd6" Jul 10 13:49:55.155903 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:55.155847 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:49:55.155903 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:55.155907 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:49:55.155903 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:55.155922 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6wdpv for pod openshift-network-diagnostics/network-check-target-hv628: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:49:55.156161 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:55.155992 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv podName:f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00 nodeName:}" failed. No retries permitted until 2026-07-10 13:49:55.655974405 +0000 UTC m=+3.119579850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6wdpv" (UniqueName: "kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv") pod "network-check-target-hv628" (UID: "f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:49:55.158912 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.158883 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwdnw\" (UniqueName: \"kubernetes.io/projected/4103f91e-397c-4638-9754-b5d9567f1f22-kube-api-access-mwdnw\") pod \"iptables-alerter-4qc7h\" (UID: \"4103f91e-397c-4638-9754-b5d9567f1f22\") " pod="openshift-network-operator/iptables-alerter-4qc7h" Jul 10 13:49:55.159129 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.159091 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55rrc\" (UniqueName: \"kubernetes.io/projected/5159ab19-b12d-4bbc-b6b1-68a19282cd6b-kube-api-access-55rrc\") pod \"multus-additional-cni-plugins-jmkqc\" (UID: \"5159ab19-b12d-4bbc-b6b1-68a19282cd6b\") " pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.159417 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.159396 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl2ff\" (UniqueName: \"kubernetes.io/projected/572667f8-51ec-40b5-8678-d1373066ffac-kube-api-access-rl2ff\") pod \"network-metrics-daemon-zpqdz\" (UID: \"572667f8-51ec-40b5-8678-d1373066ffac\") " pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:49:55.159792 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.159767 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjtdm\" (UniqueName: \"kubernetes.io/projected/868d4bcd-6eec-475b-a138-12fa9712ae87-kube-api-access-qjtdm\") pod \"node-ca-9nsfh\" (UID: \"868d4bcd-6eec-475b-a138-12fa9712ae87\") " pod="openshift-image-registry/node-ca-9nsfh" Jul 10 13:49:55.160482 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.160434 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sb28\" (UniqueName: \"kubernetes.io/projected/51d53d7f-8689-454d-a7e4-b6eec9c9c756-kube-api-access-4sb28\") pod \"multus-4v5nf\" (UID: \"51d53d7f-8689-454d-a7e4-b6eec9c9c756\") " pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.161710 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.161690 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn42f\" (UniqueName: \"kubernetes.io/projected/4e5cb550-3f73-42da-b737-5ebaae37e370-kube-api-access-hn42f\") pod \"tuned-6pk5v\" (UID: \"4e5cb550-3f73-42da-b737-5ebaae37e370\") " pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.238785 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.238740 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:49:55.257938 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.257841 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v6qp5" Jul 10 13:49:55.269126 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.269087 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" Jul 10 13:49:55.275438 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.275409 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9nsfh" Jul 10 13:49:55.283447 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.283414 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5wgd6" Jul 10 13:49:55.293368 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.293338 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jmkqc" Jul 10 13:49:55.301019 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.300994 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" Jul 10 13:49:55.311056 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.311033 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4qc7h" Jul 10 13:49:55.318075 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.318051 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4v5nf" Jul 10 13:49:55.649122 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.649088 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret\") pod \"global-pull-secret-syncer-zbx2d\" (UID: \"c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4\") " pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:49:55.649122 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.649128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs\") pod \"network-metrics-daemon-zpqdz\" (UID: \"572667f8-51ec-40b5-8678-d1373066ffac\") " pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:49:55.649308 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:55.649250 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:49:55.649308 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:55.649292 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:49:55.649386 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:55.649325 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret podName:c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4 nodeName:}" failed. No retries permitted until 2026-07-10 13:49:56.649305143 +0000 UTC m=+4.112910569 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret") pod "global-pull-secret-syncer-zbx2d" (UID: "c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:49:55.649386 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:55.649342 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs podName:572667f8-51ec-40b5-8678-d1373066ffac nodeName:}" failed. No retries permitted until 2026-07-10 13:49:56.64933417 +0000 UTC m=+4.112939595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs") pod "network-metrics-daemon-zpqdz" (UID: "572667f8-51ec-40b5-8678-d1373066ffac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:49:55.749729 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.749679 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wdpv\" (UniqueName: \"kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv\") pod \"network-check-target-hv628\" (UID: \"f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00\") " pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:49:55.749911 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:55.749888 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:49:55.749974 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:55.749919 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:49:55.749974 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:55.749931 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6wdpv for pod openshift-network-diagnostics/network-check-target-hv628: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:49:55.750035 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:55.749988 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv podName:f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00 nodeName:}" failed. No retries permitted until 2026-07-10 13:49:56.749971673 +0000 UTC m=+4.213577098 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6wdpv" (UniqueName: "kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv") pod "network-check-target-hv628" (UID: "f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:49:55.949262 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:55.949170 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 13:49:56.017391 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:56.017321 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 13:44:55 +0000 UTC" deadline="2028-02-24 11:51:20.308395825 +0000 UTC" Jul 10 13:49:56.017391 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:56.017380 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14254h1m24.291020774s" Jul 10 13:49:56.064772 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:56.064726 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5944e2_4460_42a5_9e6e_4bbbc51f60ed.slice/crio-2eb9dc90df6e3396d09d48b6456cbd7ef94f9b7275cf908a70c707db806baa45 WatchSource:0}: Error finding container 2eb9dc90df6e3396d09d48b6456cbd7ef94f9b7275cf908a70c707db806baa45: Status 404 returned error can't find the container with id 2eb9dc90df6e3396d09d48b6456cbd7ef94f9b7275cf908a70c707db806baa45 Jul 10 13:49:56.066776 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:56.066743 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868d4bcd_6eec_475b_a138_12fa9712ae87.slice/crio-da3e412a45095680cfca3a163a62dfac7f1ae8b0358c89c8ea8d70484b114a5e WatchSource:0}: Error finding container da3e412a45095680cfca3a163a62dfac7f1ae8b0358c89c8ea8d70484b114a5e: Status 404 returned error can't find the container with id da3e412a45095680cfca3a163a62dfac7f1ae8b0358c89c8ea8d70484b114a5e Jul 10 13:49:56.067476 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:56.067407 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4103f91e_397c_4638_9754_b5d9567f1f22.slice/crio-9ff7030e004badf5d6915485ba490da543aa162e76ac9c2192227321e7ff3c6c WatchSource:0}: Error finding container 9ff7030e004badf5d6915485ba490da543aa162e76ac9c2192227321e7ff3c6c: Status 404 returned error can't find the container with id 9ff7030e004badf5d6915485ba490da543aa162e76ac9c2192227321e7ff3c6c Jul 10 13:49:56.070573 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:56.070427 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d53d7f_8689_454d_a7e4_b6eec9c9c756.slice/crio-90dc2267218fa5624fe330d36250533f2912cf37d80a410db05b1bd45f353d42 WatchSource:0}: Error finding container 90dc2267218fa5624fe330d36250533f2912cf37d80a410db05b1bd45f353d42: Status 404 returned error can't find the container with id 90dc2267218fa5624fe330d36250533f2912cf37d80a410db05b1bd45f353d42 Jul 10 13:49:56.072156 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:56.072128 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5159ab19_b12d_4bbc_b6b1_68a19282cd6b.slice/crio-0135e88674df5f2cd53f2c0929540156b46e986854d001866d1eee58b22a1323 WatchSource:0}: Error finding container 0135e88674df5f2cd53f2c0929540156b46e986854d001866d1eee58b22a1323: Status 404 returned error can't find the container with id 0135e88674df5f2cd53f2c0929540156b46e986854d001866d1eee58b22a1323 Jul 10 13:49:56.074511 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:56.074400 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5cb550_3f73_42da_b737_5ebaae37e370.slice/crio-259668e78e58e0e5d49110db1df907eff2b6532660f8ca1c713a78b4285fc05a WatchSource:0}: Error finding container 259668e78e58e0e5d49110db1df907eff2b6532660f8ca1c713a78b4285fc05a: Status 404 returned error can't find the container with id 259668e78e58e0e5d49110db1df907eff2b6532660f8ca1c713a78b4285fc05a Jul 10 13:49:56.076622 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:56.076417 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea6d43a_5030_4d5b_89e7_c8dfd95b0b27.slice/crio-75c83f9e22b582bb84f2a2183a57bb103e9b50fd458ae4aacf38fc3ec81b7a27 WatchSource:0}: Error finding container 75c83f9e22b582bb84f2a2183a57bb103e9b50fd458ae4aacf38fc3ec81b7a27: Status 404 returned error can't find the container with id 75c83f9e22b582bb84f2a2183a57bb103e9b50fd458ae4aacf38fc3ec81b7a27 Jul 10 13:49:56.076869 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:56.076748 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod109c0a7d_50e6_4d3d_ac0a_fb27c253bcba.slice/crio-7c575d3b97183b93ee4f9e4e66a67d0256002549a1f44169692a3d2ef7715e44 WatchSource:0}: Error finding container 7c575d3b97183b93ee4f9e4e66a67d0256002549a1f44169692a3d2ef7715e44: Status 404 returned error can't find the container with id 7c575d3b97183b93ee4f9e4e66a67d0256002549a1f44169692a3d2ef7715e44 Jul 10 13:49:56.078545 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:49:56.078417 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3e7158_27d5_4642_b99f_7f2a8a728726.slice/crio-7e482a598bb915f0a168d866ae76b05d58c8a12f2a4501d7c9f558d303aa3fbc WatchSource:0}: Error finding container 7e482a598bb915f0a168d866ae76b05d58c8a12f2a4501d7c9f558d303aa3fbc: Status 404 returned error can't find the container with id 7e482a598bb915f0a168d866ae76b05d58c8a12f2a4501d7c9f558d303aa3fbc Jul 10 13:49:56.659914 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:56.658809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret\") pod \"global-pull-secret-syncer-zbx2d\" (UID: \"c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4\") " pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:49:56.659914 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:56.659139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs\") pod \"network-metrics-daemon-zpqdz\" (UID: \"572667f8-51ec-40b5-8678-d1373066ffac\") " pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:49:56.659914 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:56.659304 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:49:56.659914 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:56.659372 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs podName:572667f8-51ec-40b5-8678-d1373066ffac nodeName:}" failed. No retries permitted until 2026-07-10 13:49:58.659352534 +0000 UTC m=+6.122957965 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs") pod "network-metrics-daemon-zpqdz" (UID: "572667f8-51ec-40b5-8678-d1373066ffac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:49:56.659914 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:56.659799 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:49:56.659914 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:56.659858 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret podName:c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4 nodeName:}" failed. No retries permitted until 2026-07-10 13:49:58.659840383 +0000 UTC m=+6.123445812 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret") pod "global-pull-secret-syncer-zbx2d" (UID: "c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:49:56.759711 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:56.759484 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wdpv\" (UniqueName: \"kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv\") pod \"network-check-target-hv628\" (UID: \"f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00\") " pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:49:56.759711 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:56.759715 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:49:56.760006 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:56.759739 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:49:56.760006 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:56.759752 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6wdpv for pod openshift-network-diagnostics/network-check-target-hv628: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:49:56.760006 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:56.759819 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv podName:f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00 nodeName:}" failed. No retries permitted until 2026-07-10 13:49:58.759800146 +0000 UTC m=+6.223405600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6wdpv" (UniqueName: "kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv") pod "network-check-target-hv628" (UID: "f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:49:57.018366 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:57.018295 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 13:44:55 +0000 UTC" deadline="2027-12-26 00:59:45.459141421 +0000 UTC" Jul 10 13:49:57.018366 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:57.018341 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12803h9m48.44080539s" Jul 10 13:49:57.031050 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:57.031015 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:49:57.031050 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:57.031036 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:49:57.031269 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:57.031182 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv628" podUID="f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00" Jul 10 13:49:57.036022 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:57.031255 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbx2d" podUID="c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4" Jul 10 13:49:57.036186 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:57.036071 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:49:57.036285 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:57.036264 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpqdz" podUID="572667f8-51ec-40b5-8678-d1373066ffac" Jul 10 13:49:57.066326 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:57.066266 2576 generic.go:356] "Generic (PLEG): container finished" podID="a343a858d6edb11abb86474c850c3d4f" containerID="71bb2e7fca39e6d02a597a83e3d1b1a4de634c5379d66774755c953f831b0d00" exitCode=0 Jul 10 13:49:57.067035 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:57.067004 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal" event={"ID":"a343a858d6edb11abb86474c850c3d4f","Type":"ContainerDied","Data":"71bb2e7fca39e6d02a597a83e3d1b1a4de634c5379d66774755c953f831b0d00"} Jul 10 13:49:57.086043 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:57.085966 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-25.ec2.internal" event={"ID":"b15c5c2738510537812bfa649878a461","Type":"ContainerStarted","Data":"642730823dbd7dcc5d22faba70f20aa1548ee52ee591cb0ab86a8c3e517e10f5"} Jul 10 13:49:57.095341 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:57.095195 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" event={"ID":"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba","Type":"ContainerStarted","Data":"7c575d3b97183b93ee4f9e4e66a67d0256002549a1f44169692a3d2ef7715e44"} Jul 10 13:49:57.100968 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:57.100867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5wgd6" event={"ID":"aea6d43a-5030-4d5b-89e7-c8dfd95b0b27","Type":"ContainerStarted","Data":"75c83f9e22b582bb84f2a2183a57bb103e9b50fd458ae4aacf38fc3ec81b7a27"} Jul 10 13:49:57.121113 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:57.121025 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4v5nf" event={"ID":"51d53d7f-8689-454d-a7e4-b6eec9c9c756","Type":"ContainerStarted","Data":"90dc2267218fa5624fe330d36250533f2912cf37d80a410db05b1bd45f353d42"} Jul 10 13:49:57.146863 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:57.146761 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4qc7h" event={"ID":"4103f91e-397c-4638-9754-b5d9567f1f22","Type":"ContainerStarted","Data":"9ff7030e004badf5d6915485ba490da543aa162e76ac9c2192227321e7ff3c6c"} Jul 10 13:49:57.160226 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:57.159991 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v6qp5" event={"ID":"3c3e7158-27d5-4642-b99f-7f2a8a728726","Type":"ContainerStarted","Data":"7e482a598bb915f0a168d866ae76b05d58c8a12f2a4501d7c9f558d303aa3fbc"} Jul 10 13:49:57.163260 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:57.163148 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" event={"ID":"4e5cb550-3f73-42da-b737-5ebaae37e370","Type":"ContainerStarted","Data":"259668e78e58e0e5d49110db1df907eff2b6532660f8ca1c713a78b4285fc05a"} Jul 10 13:49:57.170586 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:57.170548 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmkqc" event={"ID":"5159ab19-b12d-4bbc-b6b1-68a19282cd6b","Type":"ContainerStarted","Data":"0135e88674df5f2cd53f2c0929540156b46e986854d001866d1eee58b22a1323"} Jul 10 13:49:57.190464 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:57.190364 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9nsfh" event={"ID":"868d4bcd-6eec-475b-a138-12fa9712ae87","Type":"ContainerStarted","Data":"da3e412a45095680cfca3a163a62dfac7f1ae8b0358c89c8ea8d70484b114a5e"} Jul 10 13:49:57.204694 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:57.204646 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" event={"ID":"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed","Type":"ContainerStarted","Data":"2eb9dc90df6e3396d09d48b6456cbd7ef94f9b7275cf908a70c707db806baa45"} Jul 10 13:49:58.233034 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:58.232995 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal" event={"ID":"a343a858d6edb11abb86474c850c3d4f","Type":"ContainerStarted","Data":"4aabbf696d4a39bff3303b0ece3a0b753e456c0dd26b87f5bc930fa063fec945"} Jul 10 13:49:58.251559 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:58.251493 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-25.ec2.internal" podStartSLOduration=3.251467653 podStartE2EDuration="3.251467653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:49:57.138500305 +0000 UTC m=+4.602105746" watchObservedRunningTime="2026-07-10 13:49:58.251467653 +0000 UTC m=+5.715073105" Jul 10 13:49:58.676971 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:58.676911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret\") pod \"global-pull-secret-syncer-zbx2d\" (UID: \"c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4\") " pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:49:58.676971 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:58.676969 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs\") pod \"network-metrics-daemon-zpqdz\" (UID: \"572667f8-51ec-40b5-8678-d1373066ffac\") " pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:49:58.677243 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:58.677115 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:49:58.677243 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:58.677238 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs podName:572667f8-51ec-40b5-8678-d1373066ffac nodeName:}" failed. No retries permitted until 2026-07-10 13:50:02.677205046 +0000 UTC m=+10.140810487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs") pod "network-metrics-daemon-zpqdz" (UID: "572667f8-51ec-40b5-8678-d1373066ffac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:49:58.677680 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:58.677656 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:49:58.677793 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:58.677714 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret podName:c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4 nodeName:}" failed. No retries permitted until 2026-07-10 13:50:02.677699204 +0000 UTC m=+10.141304644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret") pod "global-pull-secret-syncer-zbx2d" (UID: "c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:49:58.777595 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:58.777552 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wdpv\" (UniqueName: \"kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv\") pod \"network-check-target-hv628\" (UID: \"f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00\") " pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:49:58.777837 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:58.777816 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:49:58.777957 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:58.777866 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:49:58.777957 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:58.777881 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6wdpv for pod openshift-network-diagnostics/network-check-target-hv628: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:49:58.778075 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:58.777986 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv podName:f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00 nodeName:}" failed. No retries permitted until 2026-07-10 13:50:02.777965246 +0000 UTC m=+10.241570674 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6wdpv" (UniqueName: "kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv") pod "network-check-target-hv628" (UID: "f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:49:59.031274 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:59.030234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:49:59.031274 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:59.030453 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpqdz" podUID="572667f8-51ec-40b5-8678-d1373066ffac" Jul 10 13:49:59.031274 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:59.030901 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:49:59.031274 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:59.031016 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv628" podUID="f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00" Jul 10 13:49:59.031274 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:49:59.031098 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:49:59.031274 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:49:59.031175 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbx2d" podUID="c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4" Jul 10 13:50:01.030818 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:01.030782 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:01.031351 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:01.030783 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:01.031351 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:01.030947 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv628" podUID="f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00" Jul 10 13:50:01.031351 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:01.031048 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpqdz" podUID="572667f8-51ec-40b5-8678-d1373066ffac" Jul 10 13:50:01.031351 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:01.031091 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:01.031351 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:01.031183 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbx2d" podUID="c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4" Jul 10 13:50:02.712339 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:02.712282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret\") pod \"global-pull-secret-syncer-zbx2d\" (UID: \"c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4\") " pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:02.712339 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:02.712339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs\") pod \"network-metrics-daemon-zpqdz\" (UID: \"572667f8-51ec-40b5-8678-d1373066ffac\") " pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:02.712928 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:02.712508 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:50:02.712928 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:02.712585 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs podName:572667f8-51ec-40b5-8678-d1373066ffac nodeName:}" failed. No retries permitted until 2026-07-10 13:50:10.712564718 +0000 UTC m=+18.176170147 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs") pod "network-metrics-daemon-zpqdz" (UID: "572667f8-51ec-40b5-8678-d1373066ffac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:50:02.712928 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:02.712508 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:50:02.712928 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:02.712661 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret podName:c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4 nodeName:}" failed. No retries permitted until 2026-07-10 13:50:10.712644326 +0000 UTC m=+18.176249752 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret") pod "global-pull-secret-syncer-zbx2d" (UID: "c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:50:02.813401 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:02.813359 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wdpv\" (UniqueName: \"kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv\") pod \"network-check-target-hv628\" (UID: \"f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00\") " pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:02.813635 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:02.813606 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:50:02.813635 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:02.813638 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:50:02.813843 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:02.813653 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6wdpv for pod openshift-network-diagnostics/network-check-target-hv628: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:50:02.813843 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:02.813721 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv podName:f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00 nodeName:}" failed. No retries permitted until 2026-07-10 13:50:10.813700678 +0000 UTC m=+18.277306108 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6wdpv" (UniqueName: "kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv") pod "network-check-target-hv628" (UID: "f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:50:03.031390 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:03.031298 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:03.031390 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:03.031358 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:03.031637 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:03.031455 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:03.031637 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:03.031487 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpqdz" podUID="572667f8-51ec-40b5-8678-d1373066ffac" Jul 10 13:50:03.031637 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:03.031557 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv628" podUID="f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00" Jul 10 13:50:03.031820 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:03.031631 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbx2d" podUID="c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4" Jul 10 13:50:05.034812 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:05.034778 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:05.035328 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:05.034906 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbx2d" podUID="c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4" Jul 10 13:50:05.035328 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:05.035307 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:05.035474 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:05.035397 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpqdz" podUID="572667f8-51ec-40b5-8678-d1373066ffac" Jul 10 13:50:05.035474 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:05.035441 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:05.035583 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:05.035508 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv628" podUID="f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00" Jul 10 13:50:07.030160 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:07.030122 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:07.030160 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:07.030153 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:07.030682 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:07.030121 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:07.030682 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:07.030255 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv628" podUID="f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00" Jul 10 13:50:07.030682 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:07.030336 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbx2d" podUID="c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4" Jul 10 13:50:07.030682 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:07.030438 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpqdz" podUID="572667f8-51ec-40b5-8678-d1373066ffac" Jul 10 13:50:09.030591 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:09.030556 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:09.031071 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:09.030556 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:09.031071 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:09.030691 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv628" podUID="f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00" Jul 10 13:50:09.031071 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:09.030800 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpqdz" podUID="572667f8-51ec-40b5-8678-d1373066ffac" Jul 10 13:50:09.031071 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:09.030556 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:09.031071 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:09.030924 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbx2d" podUID="c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4" Jul 10 13:50:10.776869 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:10.776827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret\") pod \"global-pull-secret-syncer-zbx2d\" (UID: \"c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4\") " pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:10.776869 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:10.776861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs\") pod \"network-metrics-daemon-zpqdz\" (UID: \"572667f8-51ec-40b5-8678-d1373066ffac\") " pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:10.777403 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:10.776992 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:50:10.777403 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:10.777032 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:50:10.777403 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:10.777067 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret podName:c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4 nodeName:}" failed. No retries permitted until 2026-07-10 13:50:26.777048404 +0000 UTC m=+34.240653846 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret") pod "global-pull-secret-syncer-zbx2d" (UID: "c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:50:10.777403 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:10.777088 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs podName:572667f8-51ec-40b5-8678-d1373066ffac nodeName:}" failed. No retries permitted until 2026-07-10 13:50:26.777077605 +0000 UTC m=+34.240683036 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs") pod "network-metrics-daemon-zpqdz" (UID: "572667f8-51ec-40b5-8678-d1373066ffac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:50:10.877964 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:10.877933 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wdpv\" (UniqueName: \"kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv\") pod \"network-check-target-hv628\" (UID: \"f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00\") " pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:10.878139 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:10.878062 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:50:10.878139 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:10.878084 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:50:10.878139 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:10.878099 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6wdpv for pod openshift-network-diagnostics/network-check-target-hv628: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:50:10.878263 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:10.878149 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv podName:f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00 nodeName:}" failed. No retries permitted until 2026-07-10 13:50:26.878135607 +0000 UTC m=+34.341741034 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6wdpv" (UniqueName: "kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv") pod "network-check-target-hv628" (UID: "f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:50:11.030427 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:11.030344 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:11.030427 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:11.030371 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:11.030638 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:11.030344 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:11.030638 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:11.030477 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv628" podUID="f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00" Jul 10 13:50:11.030638 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:11.030569 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbx2d" podUID="c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4" Jul 10 13:50:11.030787 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:11.030708 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpqdz" podUID="572667f8-51ec-40b5-8678-d1373066ffac" Jul 10 13:50:13.031149 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:13.031103 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:13.031561 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:13.031211 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv628" podUID="f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00" Jul 10 13:50:13.031561 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:13.031245 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:13.031561 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:13.031354 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbx2d" podUID="c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4" Jul 10 13:50:13.031561 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:13.031417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:13.031561 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:13.031499 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpqdz" podUID="572667f8-51ec-40b5-8678-d1373066ffac" Jul 10 13:50:15.030942 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:15.030900 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:15.031509 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:15.030900 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:15.031509 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:15.031045 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpqdz" podUID="572667f8-51ec-40b5-8678-d1373066ffac" Jul 10 13:50:15.031509 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:15.030900 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:15.031509 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:15.031161 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbx2d" podUID="c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4" Jul 10 13:50:15.031509 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:15.031224 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv628" podUID="f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00" Jul 10 13:50:16.275203 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.274712 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v6qp5" event={"ID":"3c3e7158-27d5-4642-b99f-7f2a8a728726","Type":"ContainerStarted","Data":"ec3aef90c181926c15d23964157bdbd6f22284b247fd955b9bb16bf651b60e4a"} Jul 10 13:50:16.276560 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.276530 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" event={"ID":"4e5cb550-3f73-42da-b737-5ebaae37e370","Type":"ContainerStarted","Data":"73f46a4668da3f80d7ff285ca678f56d7d1644319bcf0556f99ac40f77a74ad0"} Jul 10 13:50:16.278169 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.278141 2576 generic.go:356] "Generic (PLEG): container finished" podID="5159ab19-b12d-4bbc-b6b1-68a19282cd6b" containerID="18fe97e1aa7920c73fe9605d9ee707d50a998908e2e4e535fba446867ce8bd35" exitCode=0 Jul 10 13:50:16.278307 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.278228 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmkqc" event={"ID":"5159ab19-b12d-4bbc-b6b1-68a19282cd6b","Type":"ContainerDied","Data":"18fe97e1aa7920c73fe9605d9ee707d50a998908e2e4e535fba446867ce8bd35"} Jul 10 13:50:16.280089 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.280065 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9nsfh" event={"ID":"868d4bcd-6eec-475b-a138-12fa9712ae87","Type":"ContainerStarted","Data":"6c0f85ba2c04f9c76da0bd1c88127c27841f75a3074402f2506c26741ff9ffed"} Jul 10 13:50:16.283624 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.283595 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" event={"ID":"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed","Type":"ContainerStarted","Data":"2b145fabcc1b891a52276daf5b7723fce2e896ce64d6f42cfbef0608b5f49a6a"} Jul 10 13:50:16.283767 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.283679 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" event={"ID":"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed","Type":"ContainerStarted","Data":"ca1eab01ed60430c9e71d7376812a34c20910be44359b733e4eb3e377d5a4162"} Jul 10 13:50:16.283767 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.283727 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" event={"ID":"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed","Type":"ContainerStarted","Data":"b63dde722c1060cafce6e60600cfd239b2edf0f221be44110ef7cd278e2e46a5"} Jul 10 13:50:16.283767 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.283762 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" event={"ID":"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed","Type":"ContainerStarted","Data":"74d1b7e066d77513a759891affeea195ac72f1b33caf4b6f1db57b94e2a3dabc"} Jul 10 13:50:16.283956 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.283797 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" event={"ID":"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed","Type":"ContainerStarted","Data":"fd64220498237b3edb555ed1770633ad01cca5cb19af54f293c5f057d11b693b"} Jul 10 13:50:16.283956 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.283850 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" event={"ID":"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed","Type":"ContainerStarted","Data":"0ea06d1a91744ec498b5de3a31a01df7010941a31d21153daf93abc416cf64c1"} Jul 10 13:50:16.285210 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.285187 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" event={"ID":"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba","Type":"ContainerStarted","Data":"642170b6e8e1b887bee807e7d588eb09269057dfd00d2bc46d64bd408f5e8991"} Jul 10 13:50:16.286704 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.286681 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5wgd6" event={"ID":"aea6d43a-5030-4d5b-89e7-c8dfd95b0b27","Type":"ContainerStarted","Data":"2ebee66106ade5f91158382163df9b01b851e1f06337a40d261cdf75818d9bbd"} Jul 10 13:50:16.288257 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.288232 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4v5nf" event={"ID":"51d53d7f-8689-454d-a7e4-b6eec9c9c756","Type":"ContainerStarted","Data":"6e602f58237e70eb4c1f33c752e9fd85be6d0e1918930769722ad49050be5071"} Jul 10 13:50:16.294869 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.294831 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-25.ec2.internal" podStartSLOduration=21.294797403 podStartE2EDuration="21.294797403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:49:58.253003145 +0000 UTC m=+5.716608598" watchObservedRunningTime="2026-07-10 13:50:16.294797403 +0000 UTC m=+23.758402860" Jul 10 13:50:16.315950 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.315899 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v6qp5" podStartSLOduration=2.926367554 podStartE2EDuration="22.31588347s" totalImagesPullingTime="19.389515916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:50:16.294561618 +0000 UTC m=+23.758167067" watchObservedRunningTime="2026-07-10 13:50:16.31588347 +0000 UTC m=+23.779488907" Jul 10 13:50:16.316281 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.316261 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9nsfh" podStartSLOduration=2.926176615 podStartE2EDuration="22.316254475s" totalImagesPullingTime="19.39007786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:50:16.315998027 +0000 UTC m=+23.779603473" watchObservedRunningTime="2026-07-10 13:50:16.316254475 +0000 UTC m=+23.779859961" Jul 10 13:50:16.347035 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.346984 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5wgd6" podStartSLOduration=2.965373846 podStartE2EDuration="22.346967215s" totalImagesPullingTime="19.381593369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:50:16.34673505 +0000 UTC m=+23.810340496" watchObservedRunningTime="2026-07-10 13:50:16.346967215 +0000 UTC m=+23.810572662" Jul 10 13:50:16.416428 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.416374 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6pk5v" podStartSLOduration=3.032198401 podStartE2EDuration="22.416352921s" totalImagesPullingTime="19.38415452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:50:16.383205969 +0000 UTC m=+23.846811418" watchObservedRunningTime="2026-07-10 13:50:16.416352921 +0000 UTC m=+23.879958367" Jul 10 13:50:16.441742 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:16.441663 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4v5nf" podStartSLOduration=3.000794771 podStartE2EDuration="22.441644816s" totalImagesPullingTime="19.440850045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:50:16.440956532 +0000 UTC m=+23.904561980" watchObservedRunningTime="2026-07-10 13:50:16.441644816 +0000 UTC m=+23.905250262" Jul 10 13:50:17.027151 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:17.027102 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 13:50:17.031072 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:17.031049 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:17.031181 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:17.031076 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:17.031181 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:17.031168 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpqdz" podUID="572667f8-51ec-40b5-8678-d1373066ffac" Jul 10 13:50:17.031292 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:17.031229 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:17.031292 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:17.031224 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv628" podUID="f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00" Jul 10 13:50:17.031399 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:17.031310 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbx2d" podUID="c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4" Jul 10 13:50:17.292367 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:17.292326 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" event={"ID":"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba","Type":"ContainerStarted","Data":"d8e3b031d77685e00f0e224349e41f9a3594aa6b7a30b8e580c635d196913e41"} Jul 10 13:50:17.294028 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:17.293998 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4qc7h" event={"ID":"4103f91e-397c-4638-9754-b5d9567f1f22","Type":"ContainerStarted","Data":"970f4fde5174bf33ef00e5396ff69358e96854778bdded2523c6dedb83dc98cf"} Jul 10 13:50:17.315498 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:17.315445 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4qc7h" podStartSLOduration=3.921635725 podStartE2EDuration="23.315427995s" totalImagesPullingTime="19.39379227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:50:17.315423737 +0000 UTC m=+24.779029184" watchObservedRunningTime="2026-07-10 13:50:17.315427995 +0000 UTC m=+24.779033440" Jul 10 13:50:17.998332 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:17.998220 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T13:50:17.027123823Z","UUID":"9c7bc695-0a36-4f37-b70b-a7474c82216d","Handler":null,"Name":"","Endpoint":""} Jul 10 13:50:18.000126 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:18.000105 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 13:50:18.000260 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:18.000144 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 13:50:18.300056 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:18.300014 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" event={"ID":"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed","Type":"ContainerStarted","Data":"22b337ed09f7e67c44f11e6173b59a878fb4ac575327bf4009e021603af38e79"} Jul 10 13:50:19.030729 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:19.030548 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:19.030903 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:19.030558 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:19.030903 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:19.030832 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpqdz" podUID="572667f8-51ec-40b5-8678-d1373066ffac" Jul 10 13:50:19.030903 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:19.030580 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:19.031024 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:19.030886 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv628" podUID="f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00" Jul 10 13:50:19.031024 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:19.031000 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbx2d" podUID="c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4" Jul 10 13:50:19.305031 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:19.304933 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" event={"ID":"109c0a7d-50e6-4d3d-ac0a-fb27c253bcba","Type":"ContainerStarted","Data":"4058d8f6a6b7266f29402b2adf3baad097c8b87a96d32bdc1b6b89b2441404e2"} Jul 10 13:50:19.499802 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:19.499764 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5wgd6" Jul 10 13:50:19.500524 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:19.500498 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5wgd6" Jul 10 13:50:19.522153 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:19.522100 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlrdd" podStartSLOduration=3.545827205 podStartE2EDuration="25.522081375s" totalImagesPullingTime="21.97625417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:49:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:50:19.328698301 +0000 UTC m=+26.792303768" watchObservedRunningTime="2026-07-10 13:50:19.522081375 +0000 UTC m=+26.985686822" Jul 10 13:50:21.030340 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:21.030153 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:21.031090 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:21.030153 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:21.031090 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:21.030429 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpqdz" podUID="572667f8-51ec-40b5-8678-d1373066ffac" Jul 10 13:50:21.031090 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:21.030153 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:21.031090 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:21.030498 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv628" podUID="f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00" Jul 10 13:50:21.031090 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:21.030564 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbx2d" podUID="c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4" Jul 10 13:50:21.311195 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:21.311087 2576 generic.go:356] "Generic (PLEG): container finished" podID="5159ab19-b12d-4bbc-b6b1-68a19282cd6b" containerID="ffa0be90b7be2badfa397924398e89ccca7c840fd40da698ef1d9847469e7cfc" exitCode=0 Jul 10 13:50:21.311195 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:21.311175 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmkqc" event={"ID":"5159ab19-b12d-4bbc-b6b1-68a19282cd6b","Type":"ContainerDied","Data":"ffa0be90b7be2badfa397924398e89ccca7c840fd40da698ef1d9847469e7cfc"} Jul 10 13:50:21.314862 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:21.314840 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" event={"ID":"2a5944e2-4460-42a5-9e6e-4bbbc51f60ed","Type":"ContainerStarted","Data":"62fd8a72ac52da7c4ff3bb30131bbced233e096fc6f946882d46f281d3ed5da3"} Jul 10 13:50:21.315159 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:21.315146 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:50:21.330467 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:21.330440 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:50:21.377367 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:21.377312 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" podStartSLOduration=7.9079066319999995 podStartE2EDuration="27.377297993s" totalImagesPullingTime="19.469391361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:50:21.375655763 +0000 UTC m=+28.839261208" watchObservedRunningTime="2026-07-10 13:50:21.377297993 +0000 UTC m=+28.840903438" Jul 10 13:50:22.317969 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:22.317941 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:50:22.318442 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:22.318087 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:50:22.332782 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:22.332755 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:50:22.561395 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:22.561370 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5wgd6" Jul 10 13:50:22.561531 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:22.561497 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 10 13:50:22.561981 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:22.561965 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5wgd6" Jul 10 13:50:23.030672 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:23.030641 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:23.030847 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:23.030730 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpqdz" podUID="572667f8-51ec-40b5-8678-d1373066ffac" Jul 10 13:50:23.030847 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:23.030734 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:23.030847 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:23.030832 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbx2d" podUID="c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4" Jul 10 13:50:23.030954 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:23.030840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:23.030988 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:23.030965 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv628" podUID="f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00" Jul 10 13:50:23.075388 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:23.075093 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zbx2d"] Jul 10 13:50:23.090493 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:23.090466 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zpqdz"] Jul 10 13:50:23.092849 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:23.092775 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hv628"] Jul 10 13:50:23.322183 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:23.322145 2576 generic.go:356] "Generic (PLEG): container finished" podID="5159ab19-b12d-4bbc-b6b1-68a19282cd6b" containerID="c2913eff23cd8f6d4e9116cf8c885dcc7a8beb44e36c129017657227c37b5fd7" exitCode=0 Jul 10 13:50:23.322638 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:23.322228 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmkqc" event={"ID":"5159ab19-b12d-4bbc-b6b1-68a19282cd6b","Type":"ContainerDied","Data":"c2913eff23cd8f6d4e9116cf8c885dcc7a8beb44e36c129017657227c37b5fd7"} Jul 10 13:50:23.322638 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:23.322302 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:23.322638 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:23.322402 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpqdz" podUID="572667f8-51ec-40b5-8678-d1373066ffac" Jul 10 13:50:23.322638 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:23.322535 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:23.322638 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:23.322632 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbx2d" podUID="c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4" Jul 10 13:50:23.322909 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:23.322685 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:23.322909 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:23.322756 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv628" podUID="f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00" Jul 10 13:50:25.031373 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:25.031116 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:25.031857 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:25.031116 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:25.031857 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:25.031407 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv628" podUID="f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00" Jul 10 13:50:25.031857 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:25.031512 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbx2d" podUID="c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4" Jul 10 13:50:25.031857 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:25.031116 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:25.031857 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:25.031626 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpqdz" podUID="572667f8-51ec-40b5-8678-d1373066ffac" Jul 10 13:50:25.328627 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:25.328541 2576 generic.go:356] "Generic (PLEG): container finished" podID="5159ab19-b12d-4bbc-b6b1-68a19282cd6b" containerID="d68664651134d9795f7b3f631b06678ce6a01b90258902f5cca600c1d7de555b" exitCode=0 Jul 10 13:50:25.328627 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:25.328609 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmkqc" event={"ID":"5159ab19-b12d-4bbc-b6b1-68a19282cd6b","Type":"ContainerDied","Data":"d68664651134d9795f7b3f631b06678ce6a01b90258902f5cca600c1d7de555b"} Jul 10 13:50:26.044147 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:26.044101 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:26.201157 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:26.201128 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:26.294619 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:26.294542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:26.372495 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:26.372458 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:26.469784 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:26.469746 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:26.700846 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:26.700798 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:26.774070 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:26.774037 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:26.800104 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:26.800070 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret\") pod \"global-pull-secret-syncer-zbx2d\" (UID: \"c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4\") " pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:26.800277 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:26.800111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs\") pod \"network-metrics-daemon-zpqdz\" (UID: \"572667f8-51ec-40b5-8678-d1373066ffac\") " pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:26.800277 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:26.800226 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:50:26.800389 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:26.800296 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs podName:572667f8-51ec-40b5-8678-d1373066ffac nodeName:}" failed. No retries permitted until 2026-07-10 13:50:58.800277491 +0000 UTC m=+66.263882931 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs") pod "network-metrics-daemon-zpqdz" (UID: "572667f8-51ec-40b5-8678-d1373066ffac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:50:26.800389 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:26.800228 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:50:26.800389 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:26.800369 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret podName:c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4 nodeName:}" failed. No retries permitted until 2026-07-10 13:50:58.800353791 +0000 UTC m=+66.263959218 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret") pod "global-pull-secret-syncer-zbx2d" (UID: "c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:50:26.901142 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:26.901099 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wdpv\" (UniqueName: \"kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv\") pod \"network-check-target-hv628\" (UID: \"f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00\") " pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:26.901318 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:26.901239 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:50:26.901318 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:26.901266 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:50:26.901318 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:26.901278 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6wdpv for pod openshift-network-diagnostics/network-check-target-hv628: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:50:26.901445 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:26.901344 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv podName:f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00 nodeName:}" failed. No retries permitted until 2026-07-10 13:50:58.901317465 +0000 UTC m=+66.364922896 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6wdpv" (UniqueName: "kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv") pod "network-check-target-hv628" (UID: "f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:50:27.030872 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:27.030783 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:27.031035 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:27.030783 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:27.031035 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:27.030926 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbx2d" podUID="c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4" Jul 10 13:50:27.031035 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:27.030783 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:27.031035 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:27.030994 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpqdz" podUID="572667f8-51ec-40b5-8678-d1373066ffac" Jul 10 13:50:27.031265 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:27.031048 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv628" podUID="f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00" Jul 10 13:50:27.120402 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:27.120350 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:27.220576 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:27.220531 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:27.835016 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:27.834981 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:27.933713 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:27.933681 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:28.273170 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.273144 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-25.ec2.internal" event="NodeReady" Jul 10 13:50:28.273707 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.273299 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 13:50:28.336904 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.336872 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qczvj"] Jul 10 13:50:28.339384 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.339368 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qczvj" Jul 10 13:50:28.342358 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.342337 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j6zkp\"" Jul 10 13:50:28.342519 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.342490 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 13:50:28.342838 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.342821 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 13:50:28.352475 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.352429 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l2hjm"] Jul 10 13:50:28.356913 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.355169 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qczvj"] Jul 10 13:50:28.356913 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.355292 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l2hjm" Jul 10 13:50:28.360440 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.360421 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 13:50:28.360552 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.360423 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9rbnh\"" Jul 10 13:50:28.360758 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.360741 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 13:50:28.361360 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.361345 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 13:50:28.375437 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.375405 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l2hjm"] Jul 10 13:50:28.514655 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.514605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c402cd2-eb3e-4833-98d9-32cfda11a80b-tmp-dir\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:50:28.514826 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.514746 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ce29e96-a26b-44e6-8c74-709f97aa2418-cert\") pod \"ingress-canary-l2hjm\" (UID: \"9ce29e96-a26b-44e6-8c74-709f97aa2418\") " pod="openshift-ingress-canary/ingress-canary-l2hjm" Jul 10 13:50:28.514826 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.514794 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c402cd2-eb3e-4833-98d9-32cfda11a80b-config-volume\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:50:28.514951 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.514830 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:50:28.514951 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.514857 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zfsj\" (UniqueName: \"kubernetes.io/projected/1c402cd2-eb3e-4833-98d9-32cfda11a80b-kube-api-access-2zfsj\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:50:28.514951 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.514884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf8sl\" (UniqueName: \"kubernetes.io/projected/9ce29e96-a26b-44e6-8c74-709f97aa2418-kube-api-access-gf8sl\") pod \"ingress-canary-l2hjm\" (UID: \"9ce29e96-a26b-44e6-8c74-709f97aa2418\") " pod="openshift-ingress-canary/ingress-canary-l2hjm" Jul 10 13:50:28.616023 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.615940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ce29e96-a26b-44e6-8c74-709f97aa2418-cert\") pod \"ingress-canary-l2hjm\" (UID: \"9ce29e96-a26b-44e6-8c74-709f97aa2418\") " pod="openshift-ingress-canary/ingress-canary-l2hjm" Jul 10 13:50:28.616023 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.615998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c402cd2-eb3e-4833-98d9-32cfda11a80b-config-volume\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:50:28.616221 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.616083 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:50:28.616221 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.616127 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zfsj\" (UniqueName: \"kubernetes.io/projected/1c402cd2-eb3e-4833-98d9-32cfda11a80b-kube-api-access-2zfsj\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:50:28.616221 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.616156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf8sl\" (UniqueName: \"kubernetes.io/projected/9ce29e96-a26b-44e6-8c74-709f97aa2418-kube-api-access-gf8sl\") pod \"ingress-canary-l2hjm\" (UID: \"9ce29e96-a26b-44e6-8c74-709f97aa2418\") " pod="openshift-ingress-canary/ingress-canary-l2hjm" Jul 10 13:50:28.616221 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.616188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c402cd2-eb3e-4833-98d9-32cfda11a80b-tmp-dir\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:50:28.616221 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:28.616206 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 13:50:28.616475 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:28.616275 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls podName:1c402cd2-eb3e-4833-98d9-32cfda11a80b nodeName:}" failed. No retries permitted until 2026-07-10 13:50:29.116254933 +0000 UTC m=+36.579860374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls") pod "dns-default-qczvj" (UID: "1c402cd2-eb3e-4833-98d9-32cfda11a80b") : secret "dns-default-metrics-tls" not found Jul 10 13:50:28.616542 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.616500 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c402cd2-eb3e-4833-98d9-32cfda11a80b-tmp-dir\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:50:28.616628 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.616600 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c402cd2-eb3e-4833-98d9-32cfda11a80b-config-volume\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:50:28.620563 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.620540 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ce29e96-a26b-44e6-8c74-709f97aa2418-cert\") pod \"ingress-canary-l2hjm\" (UID: \"9ce29e96-a26b-44e6-8c74-709f97aa2418\") " pod="openshift-ingress-canary/ingress-canary-l2hjm" Jul 10 13:50:28.628952 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.628923 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zfsj\" (UniqueName: \"kubernetes.io/projected/1c402cd2-eb3e-4833-98d9-32cfda11a80b-kube-api-access-2zfsj\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:50:28.629073 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.629024 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf8sl\" (UniqueName: \"kubernetes.io/projected/9ce29e96-a26b-44e6-8c74-709f97aa2418-kube-api-access-gf8sl\") pod \"ingress-canary-l2hjm\" (UID: \"9ce29e96-a26b-44e6-8c74-709f97aa2418\") " pod="openshift-ingress-canary/ingress-canary-l2hjm" Jul 10 13:50:28.666066 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:28.666036 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l2hjm" Jul 10 13:50:29.030478 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:29.030440 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:29.030478 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:29.030484 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:29.030723 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:29.030444 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:29.036028 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:29.035998 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 13:50:29.036363 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:29.036339 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 13:50:29.037159 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:29.037142 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 13:50:29.037265 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:29.037163 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-snqnd\"" Jul 10 13:50:29.037265 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:29.037191 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2nwfk\"" Jul 10 13:50:29.037265 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:29.037198 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 13:50:29.120172 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:29.120137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:50:29.120337 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:29.120265 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 13:50:29.120337 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:29.120336 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls podName:1c402cd2-eb3e-4833-98d9-32cfda11a80b nodeName:}" failed. No retries permitted until 2026-07-10 13:50:30.120315295 +0000 UTC m=+37.583920741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls") pod "dns-default-qczvj" (UID: "1c402cd2-eb3e-4833-98d9-32cfda11a80b") : secret "dns-default-metrics-tls" not found Jul 10 13:50:29.355213 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:29.355042 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l2hjm"] Jul 10 13:50:29.360693 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:50:29.360659 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce29e96_a26b_44e6_8c74_709f97aa2418.slice/crio-b7d6d5d28be406f45c86332169a45a2b9f0dc53801d6db262ea576f8ea000cf7 WatchSource:0}: Error finding container b7d6d5d28be406f45c86332169a45a2b9f0dc53801d6db262ea576f8ea000cf7: Status 404 returned error can't find the container with id b7d6d5d28be406f45c86332169a45a2b9f0dc53801d6db262ea576f8ea000cf7 Jul 10 13:50:30.117022 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:30.116989 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:30.126724 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:30.126692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:50:30.126900 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:30.126848 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 13:50:30.126959 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:30.126906 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls podName:1c402cd2-eb3e-4833-98d9-32cfda11a80b nodeName:}" failed. No retries permitted until 2026-07-10 13:50:32.126890653 +0000 UTC m=+39.590496077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls") pod "dns-default-qczvj" (UID: "1c402cd2-eb3e-4833-98d9-32cfda11a80b") : secret "dns-default-metrics-tls" not found Jul 10 13:50:30.216605 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:30.216569 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:30.299302 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:30.299268 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:30.340838 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:30.340787 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l2hjm" event={"ID":"9ce29e96-a26b-44e6-8c74-709f97aa2418","Type":"ContainerStarted","Data":"b7d6d5d28be406f45c86332169a45a2b9f0dc53801d6db262ea576f8ea000cf7"} Jul 10 13:50:30.344018 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:30.343988 2576 generic.go:356] "Generic (PLEG): container finished" podID="5159ab19-b12d-4bbc-b6b1-68a19282cd6b" containerID="583d35c3efddd21cd92c66d3160bb0c57bb9cec908170be8f486989a22182cff" exitCode=0 Jul 10 13:50:30.344137 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:30.344056 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmkqc" event={"ID":"5159ab19-b12d-4bbc-b6b1-68a19282cd6b","Type":"ContainerDied","Data":"583d35c3efddd21cd92c66d3160bb0c57bb9cec908170be8f486989a22182cff"} Jul 10 13:50:31.350902 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:31.350859 2576 generic.go:356] "Generic (PLEG): container finished" podID="5159ab19-b12d-4bbc-b6b1-68a19282cd6b" containerID="8747490900fdbf07f45ded388941f565f3a4e33da8cde156dda3652ad08eec97" exitCode=0 Jul 10 13:50:31.351294 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:31.350916 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmkqc" event={"ID":"5159ab19-b12d-4bbc-b6b1-68a19282cd6b","Type":"ContainerDied","Data":"8747490900fdbf07f45ded388941f565f3a4e33da8cde156dda3652ad08eec97"} Jul 10 13:50:32.143286 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:32.143082 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:50:32.143462 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:32.143246 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 13:50:32.143462 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:32.143378 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls podName:1c402cd2-eb3e-4833-98d9-32cfda11a80b nodeName:}" failed. No retries permitted until 2026-07-10 13:50:36.143364197 +0000 UTC m=+43.606969621 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls") pod "dns-default-qczvj" (UID: "1c402cd2-eb3e-4833-98d9-32cfda11a80b") : secret "dns-default-metrics-tls" not found Jul 10 13:50:32.354422 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:32.354386 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l2hjm" event={"ID":"9ce29e96-a26b-44e6-8c74-709f97aa2418","Type":"ContainerStarted","Data":"0012a4a48fe8c0da568ac0d52291c3c415ac932de2f2830c0948e05d5bb96ca5"} Jul 10 13:50:32.357659 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:32.357625 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmkqc" event={"ID":"5159ab19-b12d-4bbc-b6b1-68a19282cd6b","Type":"ContainerStarted","Data":"1c579e6c1dbedf243b432976a8016d469957865e16ae5ec4c750d377cbf4b68d"} Jul 10 13:50:32.381966 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:32.381915 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l2hjm" podStartSLOduration=2.125709553 podStartE2EDuration="4.381895861s" totalImagesPullingTime="2.256186308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:50:32.381609121 +0000 UTC m=+39.845214568" watchObservedRunningTime="2026-07-10 13:50:32.381895861 +0000 UTC m=+39.845501308" Jul 10 13:50:32.412294 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:32.412167 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jmkqc" podStartSLOduration=8.296526279 podStartE2EDuration="38.412149418s" totalImagesPullingTime="30.115623139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:49:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:50:32.411297809 +0000 UTC m=+39.874903255" watchObservedRunningTime="2026-07-10 13:50:32.412149418 +0000 UTC m=+39.875754864" Jul 10 13:50:32.564011 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:32.563979 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:32.654377 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:32.654351 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:32.717586 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:32.717518 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:33.689234 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:33.689198 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:33.808762 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:33.808730 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:36.169208 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:36.169168 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:50:36.169615 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:36.169333 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 13:50:36.169615 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:36.169400 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls podName:1c402cd2-eb3e-4833-98d9-32cfda11a80b nodeName:}" failed. No retries permitted until 2026-07-10 13:50:44.169384704 +0000 UTC m=+51.632990148 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls") pod "dns-default-qczvj" (UID: "1c402cd2-eb3e-4833-98d9-32cfda11a80b") : secret "dns-default-metrics-tls" not found Jul 10 13:50:44.227195 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:44.227156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:50:44.227669 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:44.227320 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 13:50:44.227669 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:44.227389 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls podName:1c402cd2-eb3e-4833-98d9-32cfda11a80b nodeName:}" failed. No retries permitted until 2026-07-10 13:51:00.227369882 +0000 UTC m=+67.690975319 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls") pod "dns-default-qczvj" (UID: "1c402cd2-eb3e-4833-98d9-32cfda11a80b") : secret "dns-default-metrics-tls" not found Jul 10 13:50:47.422187 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:47.422146 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:47.523292 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:47.523262 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:47.624896 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:47.624857 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:47.724011 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:47.723926 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:48.480203 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:48.480169 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:48.583636 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:48.583606 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:48.665465 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:48.665435 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:52.015303 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:52.015255 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:52.079984 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:52.079931 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:52.141281 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:52.141245 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:54.344656 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:54.344628 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r5xc7" Jul 10 13:50:54.532111 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:54.532077 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:54.636649 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:54.636572 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:54.731243 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:54.731210 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:55.544984 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:55.544950 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:58.836033 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:58.835984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret\") pod \"global-pull-secret-syncer-zbx2d\" (UID: \"c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4\") " pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:58.836033 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:58.836035 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs\") pod \"network-metrics-daemon-zpqdz\" (UID: \"572667f8-51ec-40b5-8678-d1373066ffac\") " pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:50:58.838954 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:58.838932 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 13:50:58.839023 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:58.838956 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 13:50:58.846225 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:58.846204 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 13:50:58.846303 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:50:58.846282 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs podName:572667f8-51ec-40b5-8678-d1373066ffac nodeName:}" failed. No retries permitted until 2026-07-10 13:52:02.846263864 +0000 UTC m=+130.309869289 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs") pod "network-metrics-daemon-zpqdz" (UID: "572667f8-51ec-40b5-8678-d1373066ffac") : secret "metrics-daemon-secret" not found Jul 10 13:50:58.850071 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:58.850035 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4-original-pull-secret\") pod \"global-pull-secret-syncer-zbx2d\" (UID: \"c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4\") " pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:58.937435 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:58.937382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wdpv\" (UniqueName: \"kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv\") pod \"network-check-target-hv628\" (UID: \"f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00\") " pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:58.940879 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:58.940855 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 13:50:58.951065 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:58.951030 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 13:50:58.960949 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:58.960923 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wdpv\" (UniqueName: \"kubernetes.io/projected/f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00-kube-api-access-6wdpv\") pod \"network-check-target-hv628\" (UID: \"f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00\") " pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:59.042942 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:59.042903 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbx2d" Jul 10 13:50:59.057592 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:59.057565 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-snqnd\"" Jul 10 13:50:59.065760 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:59.065728 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:50:59.184360 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:59.184331 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zbx2d"] Jul 10 13:50:59.187681 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:50:59.187656 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc708f7f5_e4fa_4dd0_9fb3_aa4d669e98f4.slice/crio-60c0b459bcb971972ab6fd98e7659580c70daf2f2fcb7578207e14f776b3ce3c WatchSource:0}: Error finding container 60c0b459bcb971972ab6fd98e7659580c70daf2f2fcb7578207e14f776b3ce3c: Status 404 returned error can't find the container with id 60c0b459bcb971972ab6fd98e7659580c70daf2f2fcb7578207e14f776b3ce3c Jul 10 13:50:59.200573 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:59.200522 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hv628"] Jul 10 13:50:59.204244 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:50:59.204219 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ffa1df_88cb_4d25_8d92_42c6d0ec6a00.slice/crio-e89bbca5d91e00628e7b18c14dd4b7db7d1c4a1d2aceb591aa28f40b6e8015d5 WatchSource:0}: Error finding container e89bbca5d91e00628e7b18c14dd4b7db7d1c4a1d2aceb591aa28f40b6e8015d5: Status 404 returned error can't find the container with id e89bbca5d91e00628e7b18c14dd4b7db7d1c4a1d2aceb591aa28f40b6e8015d5 Jul 10 13:50:59.420054 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:59.420019 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hv628" event={"ID":"f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00","Type":"ContainerStarted","Data":"e89bbca5d91e00628e7b18c14dd4b7db7d1c4a1d2aceb591aa28f40b6e8015d5"} Jul 10 13:50:59.421026 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:50:59.421001 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zbx2d" event={"ID":"c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4","Type":"ContainerStarted","Data":"60c0b459bcb971972ab6fd98e7659580c70daf2f2fcb7578207e14f776b3ce3c"} Jul 10 13:51:00.122360 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:00.122319 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:00.222933 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:00.222866 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:00.246342 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:00.246308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:51:00.246538 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:00.246478 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 13:51:00.246607 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:00.246556 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls podName:1c402cd2-eb3e-4833-98d9-32cfda11a80b nodeName:}" failed. No retries permitted until 2026-07-10 13:51:32.246535408 +0000 UTC m=+99.710140847 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls") pod "dns-default-qczvj" (UID: "1c402cd2-eb3e-4833-98d9-32cfda11a80b") : secret "dns-default-metrics-tls" not found Jul 10 13:51:00.325980 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:00.325543 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:01.958465 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:01.958429 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j"] Jul 10 13:51:01.974694 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:01.974661 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" Jul 10 13:51:01.975694 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:01.975638 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j"] Jul 10 13:51:01.985825 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:01.985786 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 13:51:01.985973 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:01.985786 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 13:51:01.986194 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:01.986174 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 10 13:51:01.988304 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:01.988280 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-79nxr\"" Jul 10 13:51:01.988432 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:01.988311 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 10 13:51:02.059055 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:02.059017 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4tm6\" (UniqueName: \"kubernetes.io/projected/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-kube-api-access-v4tm6\") pod \"cluster-monitoring-operator-5f7f494dd4-65k6j\" (UID: \"6b79eb5f-3947-4b51-b06b-f96ba7ed8e07\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" Jul 10 13:51:02.059055 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:02.059058 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-65k6j\" (UID: \"6b79eb5f-3947-4b51-b06b-f96ba7ed8e07\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" Jul 10 13:51:02.059267 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:02.059142 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-65k6j\" (UID: \"6b79eb5f-3947-4b51-b06b-f96ba7ed8e07\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" Jul 10 13:51:02.160419 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:02.160375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-65k6j\" (UID: \"6b79eb5f-3947-4b51-b06b-f96ba7ed8e07\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" Jul 10 13:51:02.160624 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:02.160454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4tm6\" (UniqueName: \"kubernetes.io/projected/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-kube-api-access-v4tm6\") pod \"cluster-monitoring-operator-5f7f494dd4-65k6j\" (UID: \"6b79eb5f-3947-4b51-b06b-f96ba7ed8e07\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" Jul 10 13:51:02.160624 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:02.160491 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-65k6j\" (UID: \"6b79eb5f-3947-4b51-b06b-f96ba7ed8e07\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" Jul 10 13:51:02.160624 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:02.160607 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 13:51:02.160831 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:02.160685 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls podName:6b79eb5f-3947-4b51-b06b-f96ba7ed8e07 nodeName:}" failed. No retries permitted until 2026-07-10 13:51:02.660663822 +0000 UTC m=+70.124269252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-65k6j" (UID: "6b79eb5f-3947-4b51-b06b-f96ba7ed8e07") : secret "cluster-monitoring-operator-tls" not found Jul 10 13:51:02.161216 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:02.161166 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-65k6j\" (UID: \"6b79eb5f-3947-4b51-b06b-f96ba7ed8e07\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" Jul 10 13:51:02.170432 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:02.170398 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:02.171722 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:02.171694 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4tm6\" (UniqueName: \"kubernetes.io/projected/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-kube-api-access-v4tm6\") pod \"cluster-monitoring-operator-5f7f494dd4-65k6j\" (UID: \"6b79eb5f-3947-4b51-b06b-f96ba7ed8e07\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" Jul 10 13:51:02.277352 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:02.277269 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:02.663966 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:02.663919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-65k6j\" (UID: \"6b79eb5f-3947-4b51-b06b-f96ba7ed8e07\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" Jul 10 13:51:02.664157 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:02.664067 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 13:51:02.664157 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:02.664146 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls podName:6b79eb5f-3947-4b51-b06b-f96ba7ed8e07 nodeName:}" failed. No retries permitted until 2026-07-10 13:51:03.664129705 +0000 UTC m=+71.127735133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-65k6j" (UID: "6b79eb5f-3947-4b51-b06b-f96ba7ed8e07") : secret "cluster-monitoring-operator-tls" not found Jul 10 13:51:03.665844 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:03.665730 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:03.672859 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:03.672794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-65k6j\" (UID: \"6b79eb5f-3947-4b51-b06b-f96ba7ed8e07\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" Jul 10 13:51:03.673038 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:03.672952 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 13:51:03.673038 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:03.673019 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls podName:6b79eb5f-3947-4b51-b06b-f96ba7ed8e07 nodeName:}" failed. No retries permitted until 2026-07-10 13:51:05.673003353 +0000 UTC m=+73.136608777 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-65k6j" (UID: "6b79eb5f-3947-4b51-b06b-f96ba7ed8e07") : secret "cluster-monitoring-operator-tls" not found Jul 10 13:51:03.766599 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:03.766516 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:03.866976 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:03.866943 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:03.942651 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:03.942617 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-whjfm"] Jul 10 13:51:03.944737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:03.944714 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-whjfm" Jul 10 13:51:03.947534 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:03.947512 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vsslg\"" Jul 10 13:51:03.948750 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:03.948730 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 10 13:51:03.948880 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:03.948751 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 10 13:51:03.957751 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:03.957727 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-whjfm"] Jul 10 13:51:03.976285 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:03.976245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw26q\" (UniqueName: \"kubernetes.io/projected/cc5b6a43-58f9-4257-a621-613436e24b9d-kube-api-access-vw26q\") pod \"volume-data-source-validator-78495597f6-whjfm\" (UID: \"cc5b6a43-58f9-4257-a621-613436e24b9d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-whjfm" Jul 10 13:51:04.077331 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.077239 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw26q\" (UniqueName: \"kubernetes.io/projected/cc5b6a43-58f9-4257-a621-613436e24b9d-kube-api-access-vw26q\") pod \"volume-data-source-validator-78495597f6-whjfm\" (UID: \"cc5b6a43-58f9-4257-a621-613436e24b9d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-whjfm" Jul 10 13:51:04.086381 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.086339 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw26q\" (UniqueName: \"kubernetes.io/projected/cc5b6a43-58f9-4257-a621-613436e24b9d-kube-api-access-vw26q\") pod \"volume-data-source-validator-78495597f6-whjfm\" (UID: \"cc5b6a43-58f9-4257-a621-613436e24b9d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-whjfm" Jul 10 13:51:04.253768 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.253717 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-whjfm" Jul 10 13:51:04.385917 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.385887 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-whjfm"] Jul 10 13:51:04.389176 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:51:04.389154 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5b6a43_58f9_4257_a621_613436e24b9d.slice/crio-5f80fa75865d1bdf56229aeb7bd5e7842bf1d424f23bfa24882eb295b971d024 WatchSource:0}: Error finding container 5f80fa75865d1bdf56229aeb7bd5e7842bf1d424f23bfa24882eb295b971d024: Status 404 returned error can't find the container with id 5f80fa75865d1bdf56229aeb7bd5e7842bf1d424f23bfa24882eb295b971d024 Jul 10 13:51:04.435870 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.435834 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-whjfm" event={"ID":"cc5b6a43-58f9-4257-a621-613436e24b9d","Type":"ContainerStarted","Data":"5f80fa75865d1bdf56229aeb7bd5e7842bf1d424f23bfa24882eb295b971d024"} Jul 10 13:51:04.437201 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.437168 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zbx2d" event={"ID":"c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4","Type":"ContainerStarted","Data":"4197b451299dc1d334201b8b7eb7fc138d6e13ac26d59c9320ae6704df5ee443"} Jul 10 13:51:04.438626 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.438605 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hv628" event={"ID":"f7ffa1df-88cb-4d25-8d92-42c6d0ec6a00","Type":"ContainerStarted","Data":"082173d8a7f01987b85fbd7a3ff939132e8e12a7e4c9bbd449f9a012ac8552d2"} Jul 10 13:51:04.438755 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.438740 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:51:04.459029 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.458975 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zbx2d" podStartSLOduration=66.134474507 podStartE2EDuration="1m10.458960039s" totalImagesPullingTime="4.324485532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:51:04.458344714 +0000 UTC m=+71.921950159" watchObservedRunningTime="2026-07-10 13:51:04.458960039 +0000 UTC m=+71.922565463" Jul 10 13:51:04.478920 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.478871 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hv628" podStartSLOduration=66.174046563 podStartE2EDuration="1m10.478843468s" totalImagesPullingTime="4.304796905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:51:04.478766651 +0000 UTC m=+71.942372097" watchObservedRunningTime="2026-07-10 13:51:04.478843468 +0000 UTC m=+71.942448918" Jul 10 13:51:04.959868 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.959833 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-vpmbq"] Jul 10 13:51:04.962165 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.962141 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:04.964818 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.964777 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 10 13:51:04.965398 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.965360 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 10 13:51:04.965997 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.965980 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 10 13:51:04.966198 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.966178 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-j5l4l\"" Jul 10 13:51:04.966518 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.966498 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 10 13:51:04.971887 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.971865 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 10 13:51:04.977222 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.977196 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-vpmbq"] Jul 10 13:51:04.983103 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.983076 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b42fcdc-6276-479f-b2d6-35b4044878f6-trusted-ca\") pod \"console-operator-7866497494-vpmbq\" (UID: \"3b42fcdc-6276-479f-b2d6-35b4044878f6\") " pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:04.983267 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.983108 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6jsm\" (UniqueName: \"kubernetes.io/projected/3b42fcdc-6276-479f-b2d6-35b4044878f6-kube-api-access-g6jsm\") pod \"console-operator-7866497494-vpmbq\" (UID: \"3b42fcdc-6276-479f-b2d6-35b4044878f6\") " pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:04.983267 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.983158 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b42fcdc-6276-479f-b2d6-35b4044878f6-config\") pod \"console-operator-7866497494-vpmbq\" (UID: \"3b42fcdc-6276-479f-b2d6-35b4044878f6\") " pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:04.983267 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:04.983195 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b42fcdc-6276-479f-b2d6-35b4044878f6-serving-cert\") pod \"console-operator-7866497494-vpmbq\" (UID: \"3b42fcdc-6276-479f-b2d6-35b4044878f6\") " pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:05.083993 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:05.083956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b42fcdc-6276-479f-b2d6-35b4044878f6-config\") pod \"console-operator-7866497494-vpmbq\" (UID: \"3b42fcdc-6276-479f-b2d6-35b4044878f6\") " pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:05.084178 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:05.084034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b42fcdc-6276-479f-b2d6-35b4044878f6-serving-cert\") pod \"console-operator-7866497494-vpmbq\" (UID: \"3b42fcdc-6276-479f-b2d6-35b4044878f6\") " pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:05.084178 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:05.084096 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b42fcdc-6276-479f-b2d6-35b4044878f6-trusted-ca\") pod \"console-operator-7866497494-vpmbq\" (UID: \"3b42fcdc-6276-479f-b2d6-35b4044878f6\") " pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:05.084178 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:05.084121 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6jsm\" (UniqueName: \"kubernetes.io/projected/3b42fcdc-6276-479f-b2d6-35b4044878f6-kube-api-access-g6jsm\") pod \"console-operator-7866497494-vpmbq\" (UID: \"3b42fcdc-6276-479f-b2d6-35b4044878f6\") " pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:05.084948 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:05.084918 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b42fcdc-6276-479f-b2d6-35b4044878f6-config\") pod \"console-operator-7866497494-vpmbq\" (UID: \"3b42fcdc-6276-479f-b2d6-35b4044878f6\") " pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:05.085309 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:05.085290 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b42fcdc-6276-479f-b2d6-35b4044878f6-trusted-ca\") pod \"console-operator-7866497494-vpmbq\" (UID: \"3b42fcdc-6276-479f-b2d6-35b4044878f6\") " pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:05.086993 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:05.086969 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b42fcdc-6276-479f-b2d6-35b4044878f6-serving-cert\") pod \"console-operator-7866497494-vpmbq\" (UID: \"3b42fcdc-6276-479f-b2d6-35b4044878f6\") " pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:05.098461 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:05.098431 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6jsm\" (UniqueName: \"kubernetes.io/projected/3b42fcdc-6276-479f-b2d6-35b4044878f6-kube-api-access-g6jsm\") pod \"console-operator-7866497494-vpmbq\" (UID: \"3b42fcdc-6276-479f-b2d6-35b4044878f6\") " pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:05.274737 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:05.274651 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:05.533543 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:05.533453 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-vpmbq"] Jul 10 13:51:05.536053 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:51:05.536023 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b42fcdc_6276_479f_b2d6_35b4044878f6.slice/crio-9cd46e3ab527339870551350ecd03fbd3d1eddbee25e0bf26a4cc0438b4717d3 WatchSource:0}: Error finding container 9cd46e3ab527339870551350ecd03fbd3d1eddbee25e0bf26a4cc0438b4717d3: Status 404 returned error can't find the container with id 9cd46e3ab527339870551350ecd03fbd3d1eddbee25e0bf26a4cc0438b4717d3 Jul 10 13:51:05.690089 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:05.690052 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-65k6j\" (UID: \"6b79eb5f-3947-4b51-b06b-f96ba7ed8e07\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" Jul 10 13:51:05.690289 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:05.690182 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 13:51:05.690289 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:05.690237 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls podName:6b79eb5f-3947-4b51-b06b-f96ba7ed8e07 nodeName:}" failed. No retries permitted until 2026-07-10 13:51:09.690222463 +0000 UTC m=+77.153827887 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-65k6j" (UID: "6b79eb5f-3947-4b51-b06b-f96ba7ed8e07") : secret "cluster-monitoring-operator-tls" not found Jul 10 13:51:05.825991 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:05.825908 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:06.449264 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:06.449225 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-whjfm" event={"ID":"cc5b6a43-58f9-4257-a621-613436e24b9d","Type":"ContainerStarted","Data":"35d1cae3c8c022776ad6788f8b8c8cf41acd75c20ef463443abb93e1fd177850"} Jul 10 13:51:06.450283 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:06.450256 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-vpmbq" event={"ID":"3b42fcdc-6276-479f-b2d6-35b4044878f6","Type":"ContainerStarted","Data":"9cd46e3ab527339870551350ecd03fbd3d1eddbee25e0bf26a4cc0438b4717d3"} Jul 10 13:51:06.468452 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:06.468397 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-whjfm" podStartSLOduration=2.43072589 podStartE2EDuration="3.468379064s" totalImagesPullingTime="1.037653174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:51:06.467707952 +0000 UTC m=+73.931313399" watchObservedRunningTime="2026-07-10 13:51:06.468379064 +0000 UTC m=+73.931984509" Jul 10 13:51:06.571494 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:06.571456 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:06.911491 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:06.911457 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n4ddz"] Jul 10 13:51:06.913509 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:06.913492 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:06.916346 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:06.916323 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 13:51:06.916346 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:06.916335 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xrxqc\"" Jul 10 13:51:06.916523 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:06.916372 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 13:51:06.917384 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:06.917353 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 13:51:06.917384 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:06.917371 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 13:51:06.928288 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:06.928260 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n4ddz"] Jul 10 13:51:06.942378 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:06.942335 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:06.999887 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:06.999857 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5fef8787-631b-4b5e-9330-15d0f084479d-crio-socket\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:07.000043 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:06.999894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fef8787-631b-4b5e-9330-15d0f084479d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:07.000043 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:06.999971 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5fef8787-631b-4b5e-9330-15d0f084479d-data-volume\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:07.000043 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:06.999992 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:07.000043 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:07.000022 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzlhp\" (UniqueName: \"kubernetes.io/projected/5fef8787-631b-4b5e-9330-15d0f084479d-kube-api-access-mzlhp\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:07.015974 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:07.015939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:07.100965 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:07.100931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5fef8787-631b-4b5e-9330-15d0f084479d-data-volume\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:07.101173 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:07.100981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:07.101173 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:07.101045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzlhp\" (UniqueName: \"kubernetes.io/projected/5fef8787-631b-4b5e-9330-15d0f084479d-kube-api-access-mzlhp\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:07.101173 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:07.101079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5fef8787-631b-4b5e-9330-15d0f084479d-crio-socket\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:07.101173 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:07.101110 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fef8787-631b-4b5e-9330-15d0f084479d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:07.101375 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:07.101181 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 13:51:07.101375 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:07.101191 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5fef8787-631b-4b5e-9330-15d0f084479d-crio-socket\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:07.101375 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:07.101199 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 13:51:07.101375 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:07.101247 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls podName:5fef8787-631b-4b5e-9330-15d0f084479d nodeName:}" failed. No retries permitted until 2026-07-10 13:51:07.601225933 +0000 UTC m=+75.064831365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n4ddz" (UID: "5fef8787-631b-4b5e-9330-15d0f084479d") : secret "insights-runtime-extractor-tls" not found Jul 10 13:51:07.101375 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:07.101271 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fef8787-631b-4b5e-9330-15d0f084479d-kube-rbac-proxy-cm podName:5fef8787-631b-4b5e-9330-15d0f084479d nodeName:}" failed. No retries permitted until 2026-07-10 13:51:07.601260565 +0000 UTC m=+75.064866003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5fef8787-631b-4b5e-9330-15d0f084479d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n4ddz" (UID: "5fef8787-631b-4b5e-9330-15d0f084479d") : configmap "kube-rbac-proxy" not found Jul 10 13:51:07.101375 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:07.101334 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5fef8787-631b-4b5e-9330-15d0f084479d-data-volume\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:07.117481 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:07.117449 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzlhp\" (UniqueName: \"kubernetes.io/projected/5fef8787-631b-4b5e-9330-15d0f084479d-kube-api-access-mzlhp\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:07.604617 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:07.604574 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fef8787-631b-4b5e-9330-15d0f084479d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:07.605155 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:07.604664 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:07.605155 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:07.604699 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 13:51:07.605155 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:07.604780 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fef8787-631b-4b5e-9330-15d0f084479d-kube-rbac-proxy-cm podName:5fef8787-631b-4b5e-9330-15d0f084479d nodeName:}" failed. No retries permitted until 2026-07-10 13:51:08.604762168 +0000 UTC m=+76.068367592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5fef8787-631b-4b5e-9330-15d0f084479d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n4ddz" (UID: "5fef8787-631b-4b5e-9330-15d0f084479d") : configmap "kube-rbac-proxy" not found Jul 10 13:51:07.605155 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:07.604783 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 13:51:07.605155 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:07.604847 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls podName:5fef8787-631b-4b5e-9330-15d0f084479d nodeName:}" failed. No retries permitted until 2026-07-10 13:51:08.604834931 +0000 UTC m=+76.068440354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n4ddz" (UID: "5fef8787-631b-4b5e-9330-15d0f084479d") : secret "insights-runtime-extractor-tls" not found Jul 10 13:51:08.456876 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:08.456846 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/0.log" Jul 10 13:51:08.457036 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:08.456884 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b42fcdc-6276-479f-b2d6-35b4044878f6" containerID="b09427798e704a1ece1610a22ade29b6f5b6dbf5d3fa8bf16a2fd45186df44e5" exitCode=255 Jul 10 13:51:08.457036 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:08.456940 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-vpmbq" event={"ID":"3b42fcdc-6276-479f-b2d6-35b4044878f6","Type":"ContainerDied","Data":"b09427798e704a1ece1610a22ade29b6f5b6dbf5d3fa8bf16a2fd45186df44e5"} Jul 10 13:51:08.457167 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:08.457155 2576 scope.go:117] "RemoveContainer" containerID="b09427798e704a1ece1610a22ade29b6f5b6dbf5d3fa8bf16a2fd45186df44e5" Jul 10 13:51:08.611325 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:08.611291 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fef8787-631b-4b5e-9330-15d0f084479d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:08.611752 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:08.611391 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:08.611752 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:08.611418 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 13:51:08.611752 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:08.611489 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fef8787-631b-4b5e-9330-15d0f084479d-kube-rbac-proxy-cm podName:5fef8787-631b-4b5e-9330-15d0f084479d nodeName:}" failed. No retries permitted until 2026-07-10 13:51:10.61147346 +0000 UTC m=+78.075078885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5fef8787-631b-4b5e-9330-15d0f084479d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n4ddz" (UID: "5fef8787-631b-4b5e-9330-15d0f084479d") : configmap "kube-rbac-proxy" not found Jul 10 13:51:08.611752 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:08.611520 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 13:51:08.611752 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:08.611576 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls podName:5fef8787-631b-4b5e-9330-15d0f084479d nodeName:}" failed. No retries permitted until 2026-07-10 13:51:10.611559853 +0000 UTC m=+78.075165279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n4ddz" (UID: "5fef8787-631b-4b5e-9330-15d0f084479d") : secret "insights-runtime-extractor-tls" not found Jul 10 13:51:08.959839 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:08.959793 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt"] Jul 10 13:51:08.961754 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:08.961739 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt" Jul 10 13:51:08.964250 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:08.964217 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 10 13:51:08.964250 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:08.964230 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 10 13:51:08.965383 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:08.965366 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jvwqh\"" Jul 10 13:51:08.965475 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:08.965390 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 10 13:51:08.965475 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:08.965442 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 10 13:51:08.974675 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:08.974647 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt"] Jul 10 13:51:09.014981 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.014936 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8jdd\" (UniqueName: \"kubernetes.io/projected/4d7b6c85-378b-4132-8b60-47b7b898dbc8-kube-api-access-m8jdd\") pod \"kube-storage-version-migrator-operator-799cb7f958-gphvt\" (UID: \"4d7b6c85-378b-4132-8b60-47b7b898dbc8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt" Jul 10 13:51:09.015165 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.014991 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4d7b6c85-378b-4132-8b60-47b7b898dbc8-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-gphvt\" (UID: \"4d7b6c85-378b-4132-8b60-47b7b898dbc8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt" Jul 10 13:51:09.015165 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.015011 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d7b6c85-378b-4132-8b60-47b7b898dbc8-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-gphvt\" (UID: \"4d7b6c85-378b-4132-8b60-47b7b898dbc8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt" Jul 10 13:51:09.084596 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.084565 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v6qp5_3c3e7158-27d5-4642-b99f-7f2a8a728726/dns-node-resolver/0.log" Jul 10 13:51:09.115521 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.115493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4d7b6c85-378b-4132-8b60-47b7b898dbc8-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-gphvt\" (UID: \"4d7b6c85-378b-4132-8b60-47b7b898dbc8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt" Jul 10 13:51:09.115619 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.115525 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d7b6c85-378b-4132-8b60-47b7b898dbc8-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-gphvt\" (UID: \"4d7b6c85-378b-4132-8b60-47b7b898dbc8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt" Jul 10 13:51:09.115619 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.115594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8jdd\" (UniqueName: \"kubernetes.io/projected/4d7b6c85-378b-4132-8b60-47b7b898dbc8-kube-api-access-m8jdd\") pod \"kube-storage-version-migrator-operator-799cb7f958-gphvt\" (UID: \"4d7b6c85-378b-4132-8b60-47b7b898dbc8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt" Jul 10 13:51:09.116143 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.116119 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d7b6c85-378b-4132-8b60-47b7b898dbc8-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-gphvt\" (UID: \"4d7b6c85-378b-4132-8b60-47b7b898dbc8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt" Jul 10 13:51:09.117698 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.117680 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4d7b6c85-378b-4132-8b60-47b7b898dbc8-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-gphvt\" (UID: \"4d7b6c85-378b-4132-8b60-47b7b898dbc8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt" Jul 10 13:51:09.126978 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.126955 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8jdd\" (UniqueName: \"kubernetes.io/projected/4d7b6c85-378b-4132-8b60-47b7b898dbc8-kube-api-access-m8jdd\") pod \"kube-storage-version-migrator-operator-799cb7f958-gphvt\" (UID: \"4d7b6c85-378b-4132-8b60-47b7b898dbc8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt" Jul 10 13:51:09.270751 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.270645 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt" Jul 10 13:51:09.395752 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.395720 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt"] Jul 10 13:51:09.398857 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:51:09.398833 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7b6c85_378b_4132_8b60_47b7b898dbc8.slice/crio-9ff61d17a4180903d7e05100bec03f85106626f9ad49e98d6fa88f133cb555cc WatchSource:0}: Error finding container 9ff61d17a4180903d7e05100bec03f85106626f9ad49e98d6fa88f133cb555cc: Status 404 returned error can't find the container with id 9ff61d17a4180903d7e05100bec03f85106626f9ad49e98d6fa88f133cb555cc Jul 10 13:51:09.461803 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.461776 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/1.log" Jul 10 13:51:09.462249 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.462231 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/0.log" Jul 10 13:51:09.462303 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.462269 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b42fcdc-6276-479f-b2d6-35b4044878f6" containerID="9a63ba5e6952a3565a10d90f92e73f43734f20bcf32f00cba9283b365860e4b3" exitCode=255 Jul 10 13:51:09.462354 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.462337 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-vpmbq" event={"ID":"3b42fcdc-6276-479f-b2d6-35b4044878f6","Type":"ContainerDied","Data":"9a63ba5e6952a3565a10d90f92e73f43734f20bcf32f00cba9283b365860e4b3"} Jul 10 13:51:09.462407 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.462395 2576 scope.go:117] "RemoveContainer" containerID="b09427798e704a1ece1610a22ade29b6f5b6dbf5d3fa8bf16a2fd45186df44e5" Jul 10 13:51:09.462642 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.462619 2576 scope.go:117] "RemoveContainer" containerID="9a63ba5e6952a3565a10d90f92e73f43734f20bcf32f00cba9283b365860e4b3" Jul 10 13:51:09.462864 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:09.462841 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-vpmbq_openshift-console-operator(3b42fcdc-6276-479f-b2d6-35b4044878f6)\"" pod="openshift-console-operator/console-operator-7866497494-vpmbq" podUID="3b42fcdc-6276-479f-b2d6-35b4044878f6" Jul 10 13:51:09.463537 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.463510 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt" event={"ID":"4d7b6c85-378b-4132-8b60-47b7b898dbc8","Type":"ContainerStarted","Data":"9ff61d17a4180903d7e05100bec03f85106626f9ad49e98d6fa88f133cb555cc"} Jul 10 13:51:09.486540 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.486516 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9nsfh_868d4bcd-6eec-475b-a138-12fa9712ae87/node-ca/0.log" Jul 10 13:51:09.719718 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:09.719681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-65k6j\" (UID: \"6b79eb5f-3947-4b51-b06b-f96ba7ed8e07\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" Jul 10 13:51:09.720105 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:09.719854 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 13:51:09.720105 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:09.719919 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls podName:6b79eb5f-3947-4b51-b06b-f96ba7ed8e07 nodeName:}" failed. No retries permitted until 2026-07-10 13:51:17.719903236 +0000 UTC m=+85.183508660 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-65k6j" (UID: "6b79eb5f-3947-4b51-b06b-f96ba7ed8e07") : secret "cluster-monitoring-operator-tls" not found Jul 10 13:51:10.468498 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:10.468464 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/1.log" Jul 10 13:51:10.468864 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:10.468845 2576 scope.go:117] "RemoveContainer" containerID="9a63ba5e6952a3565a10d90f92e73f43734f20bcf32f00cba9283b365860e4b3" Jul 10 13:51:10.469026 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:10.469000 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-vpmbq_openshift-console-operator(3b42fcdc-6276-479f-b2d6-35b4044878f6)\"" pod="openshift-console-operator/console-operator-7866497494-vpmbq" podUID="3b42fcdc-6276-479f-b2d6-35b4044878f6" Jul 10 13:51:10.627708 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:10.627668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:10.627934 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:10.627738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fef8787-631b-4b5e-9330-15d0f084479d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:10.627934 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:10.627861 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 13:51:10.627934 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:10.627893 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 13:51:10.627934 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:10.627924 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls podName:5fef8787-631b-4b5e-9330-15d0f084479d nodeName:}" failed. No retries permitted until 2026-07-10 13:51:14.627904845 +0000 UTC m=+82.091510288 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n4ddz" (UID: "5fef8787-631b-4b5e-9330-15d0f084479d") : secret "insights-runtime-extractor-tls" not found Jul 10 13:51:10.628142 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:10.627958 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fef8787-631b-4b5e-9330-15d0f084479d-kube-rbac-proxy-cm podName:5fef8787-631b-4b5e-9330-15d0f084479d nodeName:}" failed. No retries permitted until 2026-07-10 13:51:14.627939717 +0000 UTC m=+82.091545146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5fef8787-631b-4b5e-9330-15d0f084479d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n4ddz" (UID: "5fef8787-631b-4b5e-9330-15d0f084479d") : configmap "kube-rbac-proxy" not found Jul 10 13:51:10.683638 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:10.683610 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l2hjm_9ce29e96-a26b-44e6-8c74-709f97aa2418/serve-healthcheck-canary/0.log" Jul 10 13:51:12.476360 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:12.476327 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt" event={"ID":"4d7b6c85-378b-4132-8b60-47b7b898dbc8","Type":"ContainerStarted","Data":"2f14c9eeb56b98e812750482d876c43029ad2aadf9a484abb3b5b5c1185e66e0"} Jul 10 13:51:12.496011 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:12.495959 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt" podStartSLOduration=2.367090306 podStartE2EDuration="4.495941271s" totalImagesPullingTime="2.128850965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:51:12.494652338 +0000 UTC m=+79.958257804" watchObservedRunningTime="2026-07-10 13:51:12.495941271 +0000 UTC m=+79.959546717" Jul 10 13:51:12.995292 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:12.995254 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-6gfb4"] Jul 10 13:51:12.998498 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:12.998468 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-6gfb4" Jul 10 13:51:13.001846 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:13.001598 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pdhmk\"" Jul 10 13:51:13.002829 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:13.002789 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 10 13:51:13.003248 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:13.003228 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 10 13:51:13.004644 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:13.003859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:13.009640 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:13.009619 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-6gfb4"] Jul 10 13:51:13.046870 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:13.046838 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkst9\" (UniqueName: \"kubernetes.io/projected/6ffbf09b-2b2b-450d-a09d-352f2eb1fdd0-kube-api-access-rkst9\") pod \"migrator-7f59b54f6c-6gfb4\" (UID: \"6ffbf09b-2b2b-450d-a09d-352f2eb1fdd0\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-6gfb4" Jul 10 13:51:13.083601 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:13.083568 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:13.147322 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:13.147286 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkst9\" (UniqueName: \"kubernetes.io/projected/6ffbf09b-2b2b-450d-a09d-352f2eb1fdd0-kube-api-access-rkst9\") pod \"migrator-7f59b54f6c-6gfb4\" (UID: \"6ffbf09b-2b2b-450d-a09d-352f2eb1fdd0\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-6gfb4" Jul 10 13:51:13.154361 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:13.154338 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:13.157099 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:13.157071 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkst9\" (UniqueName: \"kubernetes.io/projected/6ffbf09b-2b2b-450d-a09d-352f2eb1fdd0-kube-api-access-rkst9\") pod \"migrator-7f59b54f6c-6gfb4\" (UID: \"6ffbf09b-2b2b-450d-a09d-352f2eb1fdd0\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-6gfb4" Jul 10 13:51:13.312548 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:13.312450 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-6gfb4" Jul 10 13:51:13.438089 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:13.438055 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-6gfb4"] Jul 10 13:51:13.441191 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:51:13.441164 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ffbf09b_2b2b_450d_a09d_352f2eb1fdd0.slice/crio-9b1c3b7c79a485255f3133e6b16719afbb24e582dcf5f2ebec45ef4543095288 WatchSource:0}: Error finding container 9b1c3b7c79a485255f3133e6b16719afbb24e582dcf5f2ebec45ef4543095288: Status 404 returned error can't find the container with id 9b1c3b7c79a485255f3133e6b16719afbb24e582dcf5f2ebec45ef4543095288 Jul 10 13:51:13.480190 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:13.480151 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-6gfb4" event={"ID":"6ffbf09b-2b2b-450d-a09d-352f2eb1fdd0","Type":"ContainerStarted","Data":"9b1c3b7c79a485255f3133e6b16719afbb24e582dcf5f2ebec45ef4543095288"} Jul 10 13:51:14.660489 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:14.660468 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:14.660804 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:14.660522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fef8787-631b-4b5e-9330-15d0f084479d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:14.660804 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:14.660606 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 13:51:14.660804 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:14.660654 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 13:51:14.660804 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:14.660664 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls podName:5fef8787-631b-4b5e-9330-15d0f084479d nodeName:}" failed. No retries permitted until 2026-07-10 13:51:22.660648138 +0000 UTC m=+90.124253562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n4ddz" (UID: "5fef8787-631b-4b5e-9330-15d0f084479d") : secret "insights-runtime-extractor-tls" not found Jul 10 13:51:14.660804 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:14.660712 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fef8787-631b-4b5e-9330-15d0f084479d-kube-rbac-proxy-cm podName:5fef8787-631b-4b5e-9330-15d0f084479d nodeName:}" failed. No retries permitted until 2026-07-10 13:51:22.660700078 +0000 UTC m=+90.124305502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5fef8787-631b-4b5e-9330-15d0f084479d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n4ddz" (UID: "5fef8787-631b-4b5e-9330-15d0f084479d") : configmap "kube-rbac-proxy" not found Jul 10 13:51:15.275029 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:15.274995 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:15.275189 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:15.275048 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:15.275360 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:15.275348 2576 scope.go:117] "RemoveContainer" containerID="9a63ba5e6952a3565a10d90f92e73f43734f20bcf32f00cba9283b365860e4b3" Jul 10 13:51:15.275495 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:15.275477 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-vpmbq_openshift-console-operator(3b42fcdc-6276-479f-b2d6-35b4044878f6)\"" pod="openshift-console-operator/console-operator-7866497494-vpmbq" podUID="3b42fcdc-6276-479f-b2d6-35b4044878f6" Jul 10 13:51:15.489100 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:15.489065 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-6gfb4" event={"ID":"6ffbf09b-2b2b-450d-a09d-352f2eb1fdd0","Type":"ContainerStarted","Data":"be5f70ea7dae051bda2d258039097c2fff9fea87e7f431ec1f97ff7a6d240fde"} Jul 10 13:51:15.489268 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:15.489113 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-6gfb4" event={"ID":"6ffbf09b-2b2b-450d-a09d-352f2eb1fdd0","Type":"ContainerStarted","Data":"fbf775fef5b4dbee292007dd1184ec3086c2a176478f6f57a820ed7389a36dc9"} Jul 10 13:51:15.509499 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:15.509441 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-6gfb4" podStartSLOduration=2.3050662539999998 podStartE2EDuration="3.509424969s" totalImagesPullingTime="1.204358715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:51:15.508162835 +0000 UTC m=+82.971768281" watchObservedRunningTime="2026-07-10 13:51:15.509424969 +0000 UTC m=+82.973030414" Jul 10 13:51:15.609575 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:15.609496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:15.969954 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:15.969924 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:16.409117 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:16.409083 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:16.510977 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:16.510945 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:16.770257 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:16.770174 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:16.863915 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:16.863885 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:17.784315 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:17.784280 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-65k6j\" (UID: \"6b79eb5f-3947-4b51-b06b-f96ba7ed8e07\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" Jul 10 13:51:17.784729 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:17.784448 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 13:51:17.784729 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:17.784522 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls podName:6b79eb5f-3947-4b51-b06b-f96ba7ed8e07 nodeName:}" failed. No retries permitted until 2026-07-10 13:51:33.784503182 +0000 UTC m=+101.248108606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-65k6j" (UID: "6b79eb5f-3947-4b51-b06b-f96ba7ed8e07") : secret "cluster-monitoring-operator-tls" not found Jul 10 13:51:18.288714 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.288675 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-bpxh5"] Jul 10 13:51:18.291382 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.291360 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-bpxh5" Jul 10 13:51:18.298725 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.298705 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 10 13:51:18.299799 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.299783 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 10 13:51:18.300044 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.300030 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 10 13:51:18.300097 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.300066 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 10 13:51:18.300143 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.300066 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-l7hhh\"" Jul 10 13:51:18.309455 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.309430 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-bpxh5"] Jul 10 13:51:18.331997 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.331958 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:18.389577 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.389544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d891c3dc-ef1f-4681-b1e1-7fdddbaf283d-signing-cabundle\") pod \"service-ca-79f68b9f4f-bpxh5\" (UID: \"d891c3dc-ef1f-4681-b1e1-7fdddbaf283d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bpxh5" Jul 10 13:51:18.389753 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.389584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d891c3dc-ef1f-4681-b1e1-7fdddbaf283d-signing-key\") pod \"service-ca-79f68b9f4f-bpxh5\" (UID: \"d891c3dc-ef1f-4681-b1e1-7fdddbaf283d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bpxh5" Jul 10 13:51:18.389753 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.389692 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trgrw\" (UniqueName: \"kubernetes.io/projected/d891c3dc-ef1f-4681-b1e1-7fdddbaf283d-kube-api-access-trgrw\") pod \"service-ca-79f68b9f4f-bpxh5\" (UID: \"d891c3dc-ef1f-4681-b1e1-7fdddbaf283d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bpxh5" Jul 10 13:51:18.398991 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.398959 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:18.490202 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.490170 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d891c3dc-ef1f-4681-b1e1-7fdddbaf283d-signing-cabundle\") pod \"service-ca-79f68b9f4f-bpxh5\" (UID: \"d891c3dc-ef1f-4681-b1e1-7fdddbaf283d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bpxh5" Jul 10 13:51:18.490347 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.490205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d891c3dc-ef1f-4681-b1e1-7fdddbaf283d-signing-key\") pod \"service-ca-79f68b9f4f-bpxh5\" (UID: \"d891c3dc-ef1f-4681-b1e1-7fdddbaf283d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bpxh5" Jul 10 13:51:18.490406 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.490362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trgrw\" (UniqueName: \"kubernetes.io/projected/d891c3dc-ef1f-4681-b1e1-7fdddbaf283d-kube-api-access-trgrw\") pod \"service-ca-79f68b9f4f-bpxh5\" (UID: \"d891c3dc-ef1f-4681-b1e1-7fdddbaf283d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bpxh5" Jul 10 13:51:18.490915 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.490891 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d891c3dc-ef1f-4681-b1e1-7fdddbaf283d-signing-cabundle\") pod \"service-ca-79f68b9f4f-bpxh5\" (UID: \"d891c3dc-ef1f-4681-b1e1-7fdddbaf283d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bpxh5" Jul 10 13:51:18.492722 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.492699 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d891c3dc-ef1f-4681-b1e1-7fdddbaf283d-signing-key\") pod \"service-ca-79f68b9f4f-bpxh5\" (UID: \"d891c3dc-ef1f-4681-b1e1-7fdddbaf283d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bpxh5" Jul 10 13:51:18.499828 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.499788 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trgrw\" (UniqueName: \"kubernetes.io/projected/d891c3dc-ef1f-4681-b1e1-7fdddbaf283d-kube-api-access-trgrw\") pod \"service-ca-79f68b9f4f-bpxh5\" (UID: \"d891c3dc-ef1f-4681-b1e1-7fdddbaf283d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bpxh5" Jul 10 13:51:18.599966 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.599881 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-bpxh5" Jul 10 13:51:18.733850 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:18.733799 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-bpxh5"] Jul 10 13:51:18.737071 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:51:18.737041 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd891c3dc_ef1f_4681_b1e1_7fdddbaf283d.slice/crio-7004f4ff1d5ced27041702f4ed4e04190fd83801ca63e8e9f58ca4a9df627529 WatchSource:0}: Error finding container 7004f4ff1d5ced27041702f4ed4e04190fd83801ca63e8e9f58ca4a9df627529: Status 404 returned error can't find the container with id 7004f4ff1d5ced27041702f4ed4e04190fd83801ca63e8e9f58ca4a9df627529 Jul 10 13:51:19.216828 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:19.216781 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:19.504445 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:19.504358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-bpxh5" event={"ID":"d891c3dc-ef1f-4681-b1e1-7fdddbaf283d","Type":"ContainerStarted","Data":"7004f4ff1d5ced27041702f4ed4e04190fd83801ca63e8e9f58ca4a9df627529"} Jul 10 13:51:20.152462 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:20.152422 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:21.513122 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:21.513086 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-bpxh5" event={"ID":"d891c3dc-ef1f-4681-b1e1-7fdddbaf283d","Type":"ContainerStarted","Data":"bd34651f0f15ccab1a2737a0f5296066ffd9c1b8d3e14a73c8e9bebc4967f4b2"} Jul 10 13:51:21.536129 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:21.536076 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-bpxh5" podStartSLOduration=1.586612557 podStartE2EDuration="3.536059772s" totalImagesPullingTime="1.949447215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:51:21.534919366 +0000 UTC m=+88.998524825" watchObservedRunningTime="2026-07-10 13:51:21.536059772 +0000 UTC m=+88.999665218" Jul 10 13:51:22.343416 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:22.343370 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:22.594751 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:22.594663 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:22.680425 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:22.680378 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:22.722696 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:22.722664 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fef8787-631b-4b5e-9330-15d0f084479d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:22.722921 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:22.722774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:22.722921 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:22.722911 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 13:51:22.723037 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:22.722974 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls podName:5fef8787-631b-4b5e-9330-15d0f084479d nodeName:}" failed. No retries permitted until 2026-07-10 13:51:38.722955561 +0000 UTC m=+106.186560989 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n4ddz" (UID: "5fef8787-631b-4b5e-9330-15d0f084479d") : secret "insights-runtime-extractor-tls" not found Jul 10 13:51:22.723431 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:22.723410 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fef8787-631b-4b5e-9330-15d0f084479d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:22.797425 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:22.797338 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:22.859156 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:22.859061 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:23.031131 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:23.031092 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:23.123326 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:23.123229 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:23.635763 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:23.635727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:24.432070 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:24.432035 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:24.644242 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:24.644208 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:24.831840 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:24.831741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:25.038015 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:25.037987 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:26.030524 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:26.030485 2576 scope.go:117] "RemoveContainer" containerID="9a63ba5e6952a3565a10d90f92e73f43734f20bcf32f00cba9283b365860e4b3" Jul 10 13:51:26.042467 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:26.042421 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:26.146426 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:26.146400 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:26.435694 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:26.435660 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:26.532325 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:26.532299 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 13:51:26.532725 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:26.532706 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/1.log" Jul 10 13:51:26.532828 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:26.532744 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b42fcdc-6276-479f-b2d6-35b4044878f6" containerID="66a954465e2295f5f069d2b5f0c8bbf8d3bcd169f49899f442d20a9230110b66" exitCode=255 Jul 10 13:51:26.532871 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:26.532823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-vpmbq" event={"ID":"3b42fcdc-6276-479f-b2d6-35b4044878f6","Type":"ContainerDied","Data":"66a954465e2295f5f069d2b5f0c8bbf8d3bcd169f49899f442d20a9230110b66"} Jul 10 13:51:26.532910 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:26.532877 2576 scope.go:117] "RemoveContainer" containerID="9a63ba5e6952a3565a10d90f92e73f43734f20bcf32f00cba9283b365860e4b3" Jul 10 13:51:26.533206 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:26.533187 2576 scope.go:117] "RemoveContainer" containerID="66a954465e2295f5f069d2b5f0c8bbf8d3bcd169f49899f442d20a9230110b66" Jul 10 13:51:26.533383 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:26.533364 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-vpmbq_openshift-console-operator(3b42fcdc-6276-479f-b2d6-35b4044878f6)\"" pod="openshift-console-operator/console-operator-7866497494-vpmbq" podUID="3b42fcdc-6276-479f-b2d6-35b4044878f6" Jul 10 13:51:26.640086 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:26.640052 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:26.841451 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:26.841360 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:26.927415 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:26.927384 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:27.043239 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:27.043204 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:27.537895 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:27.537868 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 13:51:27.814928 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:27.814847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:27.881250 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:27.881223 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:28.029833 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:28.029792 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:28.234072 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:28.234037 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:28.438498 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:28.438466 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:28.536985 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:28.536910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:28.833400 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:28.833304 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:29.633305 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:29.633269 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:29.844933 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:29.844898 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:30.034258 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:30.034221 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:30.441311 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:30.441279 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:31.172453 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:31.172419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:31.245093 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:31.245060 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:31.660696 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:31.660669 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:31.848298 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:31.848268 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:32.034512 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:32.034430 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:32.231661 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:32.231625 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:32.297294 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:32.297192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:51:32.299457 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:32.299430 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c402cd2-eb3e-4833-98d9-32cfda11a80b-metrics-tls\") pod \"dns-default-qczvj\" (UID: \"1c402cd2-eb3e-4833-98d9-32cfda11a80b\") " pod="openshift-dns/dns-default-qczvj" Jul 10 13:51:32.552533 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:32.552451 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j6zkp\"" Jul 10 13:51:32.560564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:32.560536 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qczvj" Jul 10 13:51:32.699643 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:32.699609 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qczvj"] Jul 10 13:51:32.702759 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:51:32.702729 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c402cd2_eb3e_4833_98d9_32cfda11a80b.slice/crio-60352dfa490f43628d5bd83d525337e3751b2798f72308546450cab7d542929a WatchSource:0}: Error finding container 60352dfa490f43628d5bd83d525337e3751b2798f72308546450cab7d542929a: Status 404 returned error can't find the container with id 60352dfa490f43628d5bd83d525337e3751b2798f72308546450cab7d542929a Jul 10 13:51:32.830248 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:32.830164 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:33.035546 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:33.035516 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:33.557756 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:33.557714 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qczvj" event={"ID":"1c402cd2-eb3e-4833-98d9-32cfda11a80b","Type":"ContainerStarted","Data":"60352dfa490f43628d5bd83d525337e3751b2798f72308546450cab7d542929a"} Jul 10 13:51:33.810910 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:33.810824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-65k6j\" (UID: \"6b79eb5f-3947-4b51-b06b-f96ba7ed8e07\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" Jul 10 13:51:33.813493 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:33.813464 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b79eb5f-3947-4b51-b06b-f96ba7ed8e07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-65k6j\" (UID: \"6b79eb5f-3947-4b51-b06b-f96ba7ed8e07\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" Jul 10 13:51:34.086580 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:34.086560 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" Jul 10 13:51:34.245280 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:34.245246 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j"] Jul 10 13:51:34.248265 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:51:34.248239 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b79eb5f_3947_4b51_b06b_f96ba7ed8e07.slice/crio-f1f0b2dfd0b946199c06f568bb44b6f67c4c611a1ef06183f4adf8b1b8e777f8 WatchSource:0}: Error finding container f1f0b2dfd0b946199c06f568bb44b6f67c4c611a1ef06183f4adf8b1b8e777f8: Status 404 returned error can't find the container with id f1f0b2dfd0b946199c06f568bb44b6f67c4c611a1ef06183f4adf8b1b8e777f8 Jul 10 13:51:34.563059 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:34.563024 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qczvj" event={"ID":"1c402cd2-eb3e-4833-98d9-32cfda11a80b","Type":"ContainerStarted","Data":"989942d515bba910e899ac4292ffee3a4ed0d7eb9f663bc3837fb21c6f68726d"} Jul 10 13:51:34.563528 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:34.563078 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qczvj" event={"ID":"1c402cd2-eb3e-4833-98d9-32cfda11a80b","Type":"ContainerStarted","Data":"0bf30cbb7f3f063d4c3b3a9f794562c8d49b835458ab4602171e739c716d48ab"} Jul 10 13:51:34.563528 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:34.563126 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qczvj" Jul 10 13:51:34.564189 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:34.564168 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" event={"ID":"6b79eb5f-3947-4b51-b06b-f96ba7ed8e07","Type":"ContainerStarted","Data":"f1f0b2dfd0b946199c06f568bb44b6f67c4c611a1ef06183f4adf8b1b8e777f8"} Jul 10 13:51:34.586795 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:34.586737 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qczvj" podStartSLOduration=65.216908403 podStartE2EDuration="1m6.586719484s" totalImagesPullingTime="1.369811081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:51:34.586015104 +0000 UTC m=+102.049620563" watchObservedRunningTime="2026-07-10 13:51:34.586719484 +0000 UTC m=+102.050324930" Jul 10 13:51:35.275189 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:35.275150 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:35.275380 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:35.275223 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:35.275616 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:35.275601 2576 scope.go:117] "RemoveContainer" containerID="66a954465e2295f5f069d2b5f0c8bbf8d3bcd169f49899f442d20a9230110b66" Jul 10 13:51:35.275827 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:51:35.275770 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-vpmbq_openshift-console-operator(3b42fcdc-6276-479f-b2d6-35b4044878f6)\"" pod="openshift-console-operator/console-operator-7866497494-vpmbq" podUID="3b42fcdc-6276-479f-b2d6-35b4044878f6" Jul 10 13:51:35.448326 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:35.448287 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hv628" Jul 10 13:51:36.573499 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:36.573465 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" event={"ID":"6b79eb5f-3947-4b51-b06b-f96ba7ed8e07","Type":"ContainerStarted","Data":"435c08b1d98ab5baf78b9274d029f506fa68b3a25fa4ce7d65593b30bf0c8414"} Jul 10 13:51:36.606372 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:36.606311 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-65k6j" podStartSLOduration=33.396390536 podStartE2EDuration="35.606292435s" totalImagesPullingTime="2.209901899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:51:36.605337642 +0000 UTC m=+104.068943088" watchObservedRunningTime="2026-07-10 13:51:36.606292435 +0000 UTC m=+104.069897881" Jul 10 13:51:36.856736 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:36.856643 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:36.964475 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:36.964426 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:37.061084 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:37.061047 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:37.178415 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:37.178385 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:38.750064 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:38.749958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:38.752204 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:38.752176 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fef8787-631b-4b5e-9330-15d0f084479d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n4ddz\" (UID: \"5fef8787-631b-4b5e-9330-15d0f084479d\") " pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:38.981847 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:38.981803 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s"] Jul 10 13:51:39.006284 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.006202 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s"] Jul 10 13:51:39.006404 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.006338 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.009079 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.009046 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 10 13:51:39.009301 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.009199 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 10 13:51:39.009549 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.009507 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 13:51:39.009766 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.009738 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 13:51:39.010241 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.010220 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 13:51:39.010332 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.010272 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 10 13:51:39.010922 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.010904 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 10 13:51:39.022427 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.022402 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n4ddz" Jul 10 13:51:39.088434 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.088403 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:39.153464 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.153433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7e02f1e3-d49b-4217-a2ff-be82f608021f-ca\") pod \"cluster-proxy-proxy-agent-6bbbc68859-wxh8s\" (UID: \"7e02f1e3-d49b-4217-a2ff-be82f608021f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.153464 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.153468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7e02f1e3-d49b-4217-a2ff-be82f608021f-hub\") pod \"cluster-proxy-proxy-agent-6bbbc68859-wxh8s\" (UID: \"7e02f1e3-d49b-4217-a2ff-be82f608021f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.153682 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.153489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48tzk\" (UniqueName: \"kubernetes.io/projected/7e02f1e3-d49b-4217-a2ff-be82f608021f-kube-api-access-48tzk\") pod \"cluster-proxy-proxy-agent-6bbbc68859-wxh8s\" (UID: \"7e02f1e3-d49b-4217-a2ff-be82f608021f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.153682 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.153580 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7e02f1e3-d49b-4217-a2ff-be82f608021f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bbbc68859-wxh8s\" (UID: \"7e02f1e3-d49b-4217-a2ff-be82f608021f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.153682 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.153631 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e02f1e3-d49b-4217-a2ff-be82f608021f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bbbc68859-wxh8s\" (UID: \"7e02f1e3-d49b-4217-a2ff-be82f608021f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.153682 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.153675 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e02f1e3-d49b-4217-a2ff-be82f608021f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bbbc68859-wxh8s\" (UID: \"7e02f1e3-d49b-4217-a2ff-be82f608021f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.165870 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.165744 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n4ddz"] Jul 10 13:51:39.168303 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:51:39.168278 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fef8787_631b_4b5e_9330_15d0f084479d.slice/crio-e81d7bc9670122b5d53373075818e945f96a41d3f962d5b49af9093b3bee0190 WatchSource:0}: Error finding container e81d7bc9670122b5d53373075818e945f96a41d3f962d5b49af9093b3bee0190: Status 404 returned error can't find the container with id e81d7bc9670122b5d53373075818e945f96a41d3f962d5b49af9093b3bee0190 Jul 10 13:51:39.176694 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.176669 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:39.255048 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.255013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e02f1e3-d49b-4217-a2ff-be82f608021f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bbbc68859-wxh8s\" (UID: \"7e02f1e3-d49b-4217-a2ff-be82f608021f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.255288 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.255057 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e02f1e3-d49b-4217-a2ff-be82f608021f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bbbc68859-wxh8s\" (UID: \"7e02f1e3-d49b-4217-a2ff-be82f608021f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.255288 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.255114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7e02f1e3-d49b-4217-a2ff-be82f608021f-ca\") pod \"cluster-proxy-proxy-agent-6bbbc68859-wxh8s\" (UID: \"7e02f1e3-d49b-4217-a2ff-be82f608021f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.255288 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.255139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7e02f1e3-d49b-4217-a2ff-be82f608021f-hub\") pod \"cluster-proxy-proxy-agent-6bbbc68859-wxh8s\" (UID: \"7e02f1e3-d49b-4217-a2ff-be82f608021f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.255288 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.255163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48tzk\" (UniqueName: \"kubernetes.io/projected/7e02f1e3-d49b-4217-a2ff-be82f608021f-kube-api-access-48tzk\") pod \"cluster-proxy-proxy-agent-6bbbc68859-wxh8s\" (UID: \"7e02f1e3-d49b-4217-a2ff-be82f608021f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.255486 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.255306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7e02f1e3-d49b-4217-a2ff-be82f608021f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bbbc68859-wxh8s\" (UID: \"7e02f1e3-d49b-4217-a2ff-be82f608021f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.255846 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.255801 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e02f1e3-d49b-4217-a2ff-be82f608021f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bbbc68859-wxh8s\" (UID: \"7e02f1e3-d49b-4217-a2ff-be82f608021f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.257679 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.257621 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7e02f1e3-d49b-4217-a2ff-be82f608021f-hub\") pod \"cluster-proxy-proxy-agent-6bbbc68859-wxh8s\" (UID: \"7e02f1e3-d49b-4217-a2ff-be82f608021f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.257767 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.257749 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e02f1e3-d49b-4217-a2ff-be82f608021f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bbbc68859-wxh8s\" (UID: \"7e02f1e3-d49b-4217-a2ff-be82f608021f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.257967 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.257949 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7e02f1e3-d49b-4217-a2ff-be82f608021f-ca\") pod \"cluster-proxy-proxy-agent-6bbbc68859-wxh8s\" (UID: \"7e02f1e3-d49b-4217-a2ff-be82f608021f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.258415 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.258394 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7e02f1e3-d49b-4217-a2ff-be82f608021f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bbbc68859-wxh8s\" (UID: \"7e02f1e3-d49b-4217-a2ff-be82f608021f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.268288 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.268263 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48tzk\" (UniqueName: \"kubernetes.io/projected/7e02f1e3-d49b-4217-a2ff-be82f608021f-kube-api-access-48tzk\") pod \"cluster-proxy-proxy-agent-6bbbc68859-wxh8s\" (UID: \"7e02f1e3-d49b-4217-a2ff-be82f608021f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.274667 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.274644 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:39.326164 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.326129 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" Jul 10 13:51:39.370774 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.370743 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:39.457348 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.457315 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s"] Jul 10 13:51:39.460571 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:51:39.460540 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e02f1e3_d49b_4217_a2ff_be82f608021f.slice/crio-7a367d1b140607760d400f1d277e40a2c505ffd7b19d9e56f8d3937f176d0f2b WatchSource:0}: Error finding container 7a367d1b140607760d400f1d277e40a2c505ffd7b19d9e56f8d3937f176d0f2b: Status 404 returned error can't find the container with id 7a367d1b140607760d400f1d277e40a2c505ffd7b19d9e56f8d3937f176d0f2b Jul 10 13:51:39.585793 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.585705 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n4ddz" event={"ID":"5fef8787-631b-4b5e-9330-15d0f084479d","Type":"ContainerStarted","Data":"e81d7bc9670122b5d53373075818e945f96a41d3f962d5b49af9093b3bee0190"} Jul 10 13:51:39.586831 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:39.586795 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" event={"ID":"7e02f1e3-d49b-4217-a2ff-be82f608021f","Type":"ContainerStarted","Data":"7a367d1b140607760d400f1d277e40a2c505ffd7b19d9e56f8d3937f176d0f2b"} Jul 10 13:51:42.814935 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:42.814892 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:42.924759 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:42.924717 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:44.571052 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:44.571024 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qczvj" Jul 10 13:51:45.211483 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:45.211443 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:45.311412 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:45.311307 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:45.612029 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:45.611945 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n4ddz" event={"ID":"5fef8787-631b-4b5e-9330-15d0f084479d","Type":"ContainerStarted","Data":"f0138dae26cc00eea2c0a2f5a4b95131e6e16863c387a4f7eeb13f4b3baeea09"} Jul 10 13:51:45.613880 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:45.613854 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" event={"ID":"7e02f1e3-d49b-4217-a2ff-be82f608021f","Type":"ContainerStarted","Data":"941cec47ea6d0d85560e9c96f45ac6999cf2d052538781ef7d70233a7579e1fb"} Jul 10 13:51:45.614000 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:45.613893 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" event={"ID":"7e02f1e3-d49b-4217-a2ff-be82f608021f","Type":"ContainerStarted","Data":"23f52ff6d638c86829ea78fc2218ddf028c4ff479c7bff63f34da51a784c8710"} Jul 10 13:51:45.614000 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:45.613906 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" event={"ID":"7e02f1e3-d49b-4217-a2ff-be82f608021f","Type":"ContainerStarted","Data":"2ccf671e0482210db37fbedf6fe29c566f2eab2a28c9c10b19c4a0efc1ce61ee"} Jul 10 13:51:45.636493 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:45.636440 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbbc68859-wxh8s" podStartSLOduration=2.039663515 podStartE2EDuration="7.636422079s" totalImagesPullingTime="5.596758564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:51:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:51:45.635669432 +0000 UTC m=+113.099274897" watchObservedRunningTime="2026-07-10 13:51:45.636422079 +0000 UTC m=+113.100027524" Jul 10 13:51:46.054223 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:46.054168 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:46.138858 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:46.138826 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:46.203717 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:46.203683 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:46.620115 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:46.620077 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n4ddz" event={"ID":"5fef8787-631b-4b5e-9330-15d0f084479d","Type":"ContainerStarted","Data":"ad4b8dcc15841910a9620ef25da60f1b575f3c076a3355e16bb2c36ad8558695"} Jul 10 13:51:47.031277 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:47.031249 2576 scope.go:117] "RemoveContainer" containerID="66a954465e2295f5f069d2b5f0c8bbf8d3bcd169f49899f442d20a9230110b66" Jul 10 13:51:47.625699 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:47.625669 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 13:51:47.626203 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:47.625728 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-vpmbq" event={"ID":"3b42fcdc-6276-479f-b2d6-35b4044878f6","Type":"ContainerStarted","Data":"6a26ce9f5e7fffccee64873fa72d8de497dd2e6d6ed7e75381930ba4f67c5885"} Jul 10 13:51:47.626203 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:47.626195 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:47.645830 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:47.645760 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-vpmbq" podStartSLOduration=41.660897676 podStartE2EDuration="43.645742199s" totalImagesPullingTime="1.984844523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:51:47.643516439 +0000 UTC m=+115.107121915" watchObservedRunningTime="2026-07-10 13:51:47.645742199 +0000 UTC m=+115.109347666" Jul 10 13:51:47.964451 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:47.964416 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-vpmbq" Jul 10 13:51:48.158723 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:48.158680 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:48.257641 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:48.257558 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:48.343562 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:48.343530 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:48.467289 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:48.467260 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:48.631829 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:48.631724 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n4ddz" event={"ID":"5fef8787-631b-4b5e-9330-15d0f084479d","Type":"ContainerStarted","Data":"512ff4ae56c3b294ccfa6af38388bc2d6b008ee5d2df515606cc715aa6a1821f"} Jul 10 13:51:48.636834 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:48.636794 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:48.655931 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:48.655887 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n4ddz" podStartSLOduration=33.501767909 podStartE2EDuration="42.655871406s" totalImagesPullingTime="9.154103497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:51:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:51:48.654706324 +0000 UTC m=+116.118311781" watchObservedRunningTime="2026-07-10 13:51:48.655871406 +0000 UTC m=+116.119476852" Jul 10 13:51:48.883770 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:48.883685 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:49.033086 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:49.033057 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:49.109548 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:49.109516 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:49.243085 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:49.243053 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:49.429546 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:49.429464 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:49.497770 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:49.497691 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:49.644614 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:49.644578 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:49.845747 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:49.845664 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:49.960115 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:49.960083 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:51.062149 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.062115 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:51.169790 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.169759 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:51.272404 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.272373 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:51.367701 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.367622 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:51.466185 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.466154 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:51.584919 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.584884 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gqz9s"] Jul 10 13:51:51.588399 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.588375 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.593294 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.593267 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zk5vq\"" Jul 10 13:51:51.593437 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.593352 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 13:51:51.593545 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.593528 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 13:51:51.594293 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.594277 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 13:51:51.594372 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.594296 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 13:51:51.599638 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.599611 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:51.683914 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.683882 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:51.759892 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.759859 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e8b6f70-5e39-44e5-8475-803fe93dd246-metrics-client-ca\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.760069 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.759910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1e8b6f70-5e39-44e5-8475-803fe93dd246-root\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.760069 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.760005 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1e8b6f70-5e39-44e5-8475-803fe93dd246-node-exporter-wtmp\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.760069 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.760037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e8b6f70-5e39-44e5-8475-803fe93dd246-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.760069 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.760058 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1e8b6f70-5e39-44e5-8475-803fe93dd246-node-exporter-tls\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.760252 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.760088 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1e8b6f70-5e39-44e5-8475-803fe93dd246-node-exporter-textfile\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.760252 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.760112 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1e8b6f70-5e39-44e5-8475-803fe93dd246-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.760252 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.760138 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdz44\" (UniqueName: \"kubernetes.io/projected/1e8b6f70-5e39-44e5-8475-803fe93dd246-kube-api-access-vdz44\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.760252 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.760182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e8b6f70-5e39-44e5-8475-803fe93dd246-sys\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.780946 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.780915 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:51.860943 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.860911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1e8b6f70-5e39-44e5-8475-803fe93dd246-node-exporter-wtmp\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.860943 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.860948 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e8b6f70-5e39-44e5-8475-803fe93dd246-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.861207 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.860968 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1e8b6f70-5e39-44e5-8475-803fe93dd246-node-exporter-tls\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.861207 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.860987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1e8b6f70-5e39-44e5-8475-803fe93dd246-node-exporter-textfile\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.861207 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.861125 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1e8b6f70-5e39-44e5-8475-803fe93dd246-node-exporter-wtmp\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.861207 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.861184 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1e8b6f70-5e39-44e5-8475-803fe93dd246-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.861410 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.861226 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdz44\" (UniqueName: \"kubernetes.io/projected/1e8b6f70-5e39-44e5-8475-803fe93dd246-kube-api-access-vdz44\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.861410 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.861267 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e8b6f70-5e39-44e5-8475-803fe93dd246-sys\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.861410 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.861313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e8b6f70-5e39-44e5-8475-803fe93dd246-metrics-client-ca\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.861410 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.861323 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1e8b6f70-5e39-44e5-8475-803fe93dd246-node-exporter-textfile\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.861410 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.861356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1e8b6f70-5e39-44e5-8475-803fe93dd246-root\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.861649 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.861434 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1e8b6f70-5e39-44e5-8475-803fe93dd246-root\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.861649 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.861472 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e8b6f70-5e39-44e5-8475-803fe93dd246-sys\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.861961 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.861939 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1e8b6f70-5e39-44e5-8475-803fe93dd246-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.862103 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.862082 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e8b6f70-5e39-44e5-8475-803fe93dd246-metrics-client-ca\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.863493 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.863464 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e8b6f70-5e39-44e5-8475-803fe93dd246-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.863731 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.863712 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1e8b6f70-5e39-44e5-8475-803fe93dd246-node-exporter-tls\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.872771 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.872742 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdz44\" (UniqueName: \"kubernetes.io/projected/1e8b6f70-5e39-44e5-8475-803fe93dd246-kube-api-access-vdz44\") pod \"node-exporter-gqz9s\" (UID: \"1e8b6f70-5e39-44e5-8475-803fe93dd246\") " pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.883222 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.883196 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:51.899794 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.899765 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gqz9s" Jul 10 13:51:51.911106 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:51:51.911075 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8b6f70_5e39_44e5_8475_803fe93dd246.slice/crio-94dd6036d46dcf794f70036cd5d68c5b8d511bc75fe4762770fb2fb3eea08477 WatchSource:0}: Error finding container 94dd6036d46dcf794f70036cd5d68c5b8d511bc75fe4762770fb2fb3eea08477: Status 404 returned error can't find the container with id 94dd6036d46dcf794f70036cd5d68c5b8d511bc75fe4762770fb2fb3eea08477 Jul 10 13:51:51.984712 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:51.984635 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:52.083442 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:52.083408 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:52.181593 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:52.181563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:52.281976 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:52.281902 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:52.461461 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:52.461430 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:52.562158 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:52.562078 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:52.650551 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:52.650514 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqz9s" event={"ID":"1e8b6f70-5e39-44e5-8475-803fe93dd246","Type":"ContainerStarted","Data":"94dd6036d46dcf794f70036cd5d68c5b8d511bc75fe4762770fb2fb3eea08477"} Jul 10 13:51:52.773587 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:52.773556 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:53.655041 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:53.655003 2576 generic.go:356] "Generic (PLEG): container finished" podID="1e8b6f70-5e39-44e5-8475-803fe93dd246" containerID="f1e1a2c6ade6ca8ba3035497e092ee64ba6c3321a6e05846f8a8566f13288ffe" exitCode=0 Jul 10 13:51:53.655041 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:53.655045 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqz9s" event={"ID":"1e8b6f70-5e39-44e5-8475-803fe93dd246","Type":"ContainerDied","Data":"f1e1a2c6ade6ca8ba3035497e092ee64ba6c3321a6e05846f8a8566f13288ffe"} Jul 10 13:51:54.568843 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:54.568783 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:54.645921 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:54.645889 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:54.661726 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:54.661692 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqz9s" event={"ID":"1e8b6f70-5e39-44e5-8475-803fe93dd246","Type":"ContainerStarted","Data":"1ed62ce7ffd3d6be065f82590a66b75a1bf423ea0e8ee5fa01ce2bb429fe1a05"} Jul 10 13:51:54.662125 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:54.661744 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqz9s" event={"ID":"1e8b6f70-5e39-44e5-8475-803fe93dd246","Type":"ContainerStarted","Data":"2c09417b5167dea62b28a1816d9b8b2c2bc2ff751bde692e9db4cd4e4d3603e0"} Jul 10 13:51:54.684565 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:54.684506 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gqz9s" podStartSLOduration=2.732357613 podStartE2EDuration="3.684489052s" totalImagesPullingTime="886.512156ms" totalInitContainerRuntime="65.619283ms" isStatefulPod=true podCreationTimestamp="2026-07-10 13:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:51:54.682473379 +0000 UTC m=+122.146078825" watchObservedRunningTime="2026-07-10 13:51:54.684489052 +0000 UTC m=+122.148094528" Jul 10 13:51:54.719124 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:54.719080 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:55.131586 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:55.131553 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:56.247490 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:56.247454 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:56.319891 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:56.319859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:56.824671 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:56.824638 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:57.859440 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:57.859409 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:58.009200 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:58.009168 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:58.099504 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:51:58.099469 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:01.247281 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:01.247249 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:01.318132 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:01.318101 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:01.662470 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:01.662438 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:01.805431 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:01.805401 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:01.890209 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:01.890179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:02.704113 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:02.704081 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:02.810053 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:02.810021 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:02.944886 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:02.944848 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs\") pod \"network-metrics-daemon-zpqdz\" (UID: \"572667f8-51ec-40b5-8678-d1373066ffac\") " pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:52:02.947172 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:02.947155 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/572667f8-51ec-40b5-8678-d1373066ffac-metrics-certs\") pod \"network-metrics-daemon-zpqdz\" (UID: \"572667f8-51ec-40b5-8678-d1373066ffac\") " pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:52:02.951946 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:02.951926 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2nwfk\"" Jul 10 13:52:02.959337 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:02.959286 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpqdz" Jul 10 13:52:03.088362 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:03.088329 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zpqdz"] Jul 10 13:52:03.091195 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:52:03.091159 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572667f8_51ec_40b5_8678_d1373066ffac.slice/crio-a02a5bcfed92c4ffc4db7330c583dacfc18499acddd5422e06963f8c07b67453 WatchSource:0}: Error finding container a02a5bcfed92c4ffc4db7330c583dacfc18499acddd5422e06963f8c07b67453: Status 404 returned error can't find the container with id a02a5bcfed92c4ffc4db7330c583dacfc18499acddd5422e06963f8c07b67453 Jul 10 13:52:03.692384 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:03.692346 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zpqdz" event={"ID":"572667f8-51ec-40b5-8678-d1373066ffac","Type":"ContainerStarted","Data":"a02a5bcfed92c4ffc4db7330c583dacfc18499acddd5422e06963f8c07b67453"} Jul 10 13:52:04.258027 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:04.257939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:04.338590 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:04.338557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:04.697108 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:04.697076 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zpqdz" event={"ID":"572667f8-51ec-40b5-8678-d1373066ffac","Type":"ContainerStarted","Data":"1bfb8b3c81ee8298a443dbf1cac375068f3a7559606ed53c1724d5d43580e90d"} Jul 10 13:52:04.697278 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:04.697158 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zpqdz" event={"ID":"572667f8-51ec-40b5-8678-d1373066ffac","Type":"ContainerStarted","Data":"e5ab0eb846d4b5b2a7c7d95b0f31ce8f7168823d6bb7a5a61132182c60a61c99"} Jul 10 13:52:04.716460 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:04.716409 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zpqdz" podStartSLOduration=129.824101076 podStartE2EDuration="2m10.716394353s" totalImagesPullingTime="892.293277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:52:04.71595352 +0000 UTC m=+132.179558967" watchObservedRunningTime="2026-07-10 13:52:04.716394353 +0000 UTC m=+132.179999832" Jul 10 13:52:04.849231 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:04.849197 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:04.944531 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:04.944500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:05.167320 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:05.167290 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:05.271391 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:05.271352 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:05.877982 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:05.877952 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:05.976033 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:05.975997 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:08.596001 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:08.595964 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:08.694895 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:08.694866 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:10.896205 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:10.896172 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:10.997858 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:10.997803 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:11.394034 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:11.394004 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:11.466114 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:11.466081 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:12.437441 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:12.437410 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:12.507665 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:12.507629 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:12.818326 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:12.818248 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:12.921184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:12.921154 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:13.311766 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:13.311736 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:13.383208 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:13.383174 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:20.169073 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:20.169042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:20.293100 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:20.293067 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:20.379302 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:20.379278 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:22.761658 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:22.761628 2576 generic.go:356] "Generic (PLEG): container finished" podID="4d7b6c85-378b-4132-8b60-47b7b898dbc8" containerID="2f14c9eeb56b98e812750482d876c43029ad2aadf9a484abb3b5b5c1185e66e0" exitCode=0 Jul 10 13:52:22.762074 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:22.761707 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt" event={"ID":"4d7b6c85-378b-4132-8b60-47b7b898dbc8","Type":"ContainerDied","Data":"2f14c9eeb56b98e812750482d876c43029ad2aadf9a484abb3b5b5c1185e66e0"} Jul 10 13:52:22.762074 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:22.762019 2576 scope.go:117] "RemoveContainer" containerID="2f14c9eeb56b98e812750482d876c43029ad2aadf9a484abb3b5b5c1185e66e0" Jul 10 13:52:23.766877 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:23.766840 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gphvt" event={"ID":"4d7b6c85-378b-4132-8b60-47b7b898dbc8","Type":"ContainerStarted","Data":"7ee52ded6c745f3cf4db6e556381f5d371bb3b177e1a1df5e57534bf6108236b"} Jul 10 13:52:27.830948 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:27.830916 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:27.900830 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:52:27.900764 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:53:07.707395 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:53:07.707361 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:53:07.808125 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:53:07.808094 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:53:17.101072 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:53:17.101041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:53:19.720131 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:53:19.720095 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:53:19.801699 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:53:19.801674 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:53:20.938566 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:53:20.938535 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:53:27.710801 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:53:27.710768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:53:27.811395 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:53:27.811361 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:53:27.912521 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:53:27.912488 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:53:27.972709 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:53:27.972640 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:54:07.650287 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:54:07.650251 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:54:07.711972 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:54:07.711934 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:54:17.759289 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:54:17.759216 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:54:32.706233 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:54:32.706184 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:54:32.805164 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:54:32.805088 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:54:52.932156 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:54:52.932124 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 13:54:52.934416 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:54:52.934396 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 13:54:52.942282 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:54:52.942263 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 13:54:59.625830 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:54:59.625780 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:00.993787 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:00.993751 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:03.246959 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:03.246924 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:03.345187 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:03.345154 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:04.366316 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:04.366281 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:04.478892 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:04.478862 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:05.493707 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:05.493676 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:05.599884 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:05.599855 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:07.517872 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:07.517837 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:07.904218 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:07.904193 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:09.031908 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:09.031877 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:09.164474 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:09.164442 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:10.231148 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:10.231117 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:10.294257 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:10.294220 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:16.113709 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:16.113671 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:16.793938 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:16.793906 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bl54r"] Jul 10 13:55:16.796804 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:16.796780 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bl54r" Jul 10 13:55:16.799590 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:16.799572 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 13:55:16.800632 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:16.800613 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-n9k45\"" Jul 10 13:55:16.800711 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:16.800621 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 13:55:16.806645 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:16.806622 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bl54r"] Jul 10 13:55:16.852890 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:16.852829 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zv79\" (UniqueName: \"kubernetes.io/projected/43fc5616-b9f6-4ae6-98a2-4f2d70bbc1d6-kube-api-access-8zv79\") pod \"cert-manager-55b4d954d5-bl54r\" (UID: \"43fc5616-b9f6-4ae6-98a2-4f2d70bbc1d6\") " pod="cert-manager/cert-manager-55b4d954d5-bl54r" Jul 10 13:55:16.853078 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:16.852924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/43fc5616-b9f6-4ae6-98a2-4f2d70bbc1d6-bound-sa-token\") pod \"cert-manager-55b4d954d5-bl54r\" (UID: \"43fc5616-b9f6-4ae6-98a2-4f2d70bbc1d6\") " pod="cert-manager/cert-manager-55b4d954d5-bl54r" Jul 10 13:55:16.954130 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:16.954092 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/43fc5616-b9f6-4ae6-98a2-4f2d70bbc1d6-bound-sa-token\") pod \"cert-manager-55b4d954d5-bl54r\" (UID: \"43fc5616-b9f6-4ae6-98a2-4f2d70bbc1d6\") " pod="cert-manager/cert-manager-55b4d954d5-bl54r" Jul 10 13:55:16.954299 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:16.954145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zv79\" (UniqueName: \"kubernetes.io/projected/43fc5616-b9f6-4ae6-98a2-4f2d70bbc1d6-kube-api-access-8zv79\") pod \"cert-manager-55b4d954d5-bl54r\" (UID: \"43fc5616-b9f6-4ae6-98a2-4f2d70bbc1d6\") " pod="cert-manager/cert-manager-55b4d954d5-bl54r" Jul 10 13:55:16.963409 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:16.963382 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/43fc5616-b9f6-4ae6-98a2-4f2d70bbc1d6-bound-sa-token\") pod \"cert-manager-55b4d954d5-bl54r\" (UID: \"43fc5616-b9f6-4ae6-98a2-4f2d70bbc1d6\") " pod="cert-manager/cert-manager-55b4d954d5-bl54r" Jul 10 13:55:16.963545 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:16.963528 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zv79\" (UniqueName: \"kubernetes.io/projected/43fc5616-b9f6-4ae6-98a2-4f2d70bbc1d6-kube-api-access-8zv79\") pod \"cert-manager-55b4d954d5-bl54r\" (UID: \"43fc5616-b9f6-4ae6-98a2-4f2d70bbc1d6\") " pod="cert-manager/cert-manager-55b4d954d5-bl54r" Jul 10 13:55:17.105696 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:17.105607 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bl54r" Jul 10 13:55:17.237140 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:17.237089 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bl54r"] Jul 10 13:55:17.239562 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:55:17.239537 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43fc5616_b9f6_4ae6_98a2_4f2d70bbc1d6.slice/crio-5940649edd2cab5bf5c23e0a07963b31479b76e6ed6de1570a11cbd2fe3491b0 WatchSource:0}: Error finding container 5940649edd2cab5bf5c23e0a07963b31479b76e6ed6de1570a11cbd2fe3491b0: Status 404 returned error can't find the container with id 5940649edd2cab5bf5c23e0a07963b31479b76e6ed6de1570a11cbd2fe3491b0 Jul 10 13:55:17.241488 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:17.241471 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 13:55:17.362964 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:17.362881 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bl54r" event={"ID":"43fc5616-b9f6-4ae6-98a2-4f2d70bbc1d6","Type":"ContainerStarted","Data":"5940649edd2cab5bf5c23e0a07963b31479b76e6ed6de1570a11cbd2fe3491b0"} Jul 10 13:55:20.377709 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:20.377623 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bl54r" event={"ID":"43fc5616-b9f6-4ae6-98a2-4f2d70bbc1d6","Type":"ContainerStarted","Data":"d609f60492b5441f33bb951a7a2019064e187e082961ff36c162ea93a74e0d9f"} Jul 10 13:55:20.395325 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:20.395279 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-bl54r" podStartSLOduration=1.645557294 podStartE2EDuration="4.39526487s" totalImagesPullingTime="2.749707576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:55:20.394535482 +0000 UTC m=+327.858140944" watchObservedRunningTime="2026-07-10 13:55:20.39526487 +0000 UTC m=+327.858870314" Jul 10 13:55:21.046199 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:21.046165 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:21.144281 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:21.144255 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:23.065186 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:23.065154 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cqp6x"] Jul 10 13:55:23.068297 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:23.068277 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cqp6x" Jul 10 13:55:23.070616 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:23.070597 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-264sf\"" Jul 10 13:55:23.070771 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:23.070626 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 13:55:23.071636 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:23.071621 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 13:55:23.077864 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:23.077721 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cqp6x"] Jul 10 13:55:23.106000 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:23.105967 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95z54\" (UniqueName: \"kubernetes.io/projected/92467d01-6dac-4288-9232-70b7456ab773-kube-api-access-95z54\") pod \"openshift-lws-operator-fd8ccff4c-cqp6x\" (UID: \"92467d01-6dac-4288-9232-70b7456ab773\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cqp6x" Jul 10 13:55:23.106144 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:23.106035 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92467d01-6dac-4288-9232-70b7456ab773-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cqp6x\" (UID: \"92467d01-6dac-4288-9232-70b7456ab773\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cqp6x" Jul 10 13:55:23.207013 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:23.206978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92467d01-6dac-4288-9232-70b7456ab773-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cqp6x\" (UID: \"92467d01-6dac-4288-9232-70b7456ab773\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cqp6x" Jul 10 13:55:23.207176 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:23.207028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95z54\" (UniqueName: \"kubernetes.io/projected/92467d01-6dac-4288-9232-70b7456ab773-kube-api-access-95z54\") pod \"openshift-lws-operator-fd8ccff4c-cqp6x\" (UID: \"92467d01-6dac-4288-9232-70b7456ab773\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cqp6x" Jul 10 13:55:23.207361 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:23.207340 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92467d01-6dac-4288-9232-70b7456ab773-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cqp6x\" (UID: \"92467d01-6dac-4288-9232-70b7456ab773\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cqp6x" Jul 10 13:55:23.215276 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:23.215246 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95z54\" (UniqueName: \"kubernetes.io/projected/92467d01-6dac-4288-9232-70b7456ab773-kube-api-access-95z54\") pod \"openshift-lws-operator-fd8ccff4c-cqp6x\" (UID: \"92467d01-6dac-4288-9232-70b7456ab773\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cqp6x" Jul 10 13:55:23.383829 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:23.383713 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cqp6x" Jul 10 13:55:23.511606 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:23.511574 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cqp6x"] Jul 10 13:55:23.514359 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:55:23.514334 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92467d01_6dac_4288_9232_70b7456ab773.slice/crio-cf609182cbdbb690cbaea0467f1ad0265c0b3312a24ebbebfd5ee3d9ecfd416c WatchSource:0}: Error finding container cf609182cbdbb690cbaea0467f1ad0265c0b3312a24ebbebfd5ee3d9ecfd416c: Status 404 returned error can't find the container with id cf609182cbdbb690cbaea0467f1ad0265c0b3312a24ebbebfd5ee3d9ecfd416c Jul 10 13:55:24.392397 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:24.392358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cqp6x" event={"ID":"92467d01-6dac-4288-9232-70b7456ab773","Type":"ContainerStarted","Data":"cf609182cbdbb690cbaea0467f1ad0265c0b3312a24ebbebfd5ee3d9ecfd416c"} Jul 10 13:55:27.406279 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:27.406246 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cqp6x" event={"ID":"92467d01-6dac-4288-9232-70b7456ab773","Type":"ContainerStarted","Data":"d762dcb123663f43872477e03ff52245dde00a4c67d61330d0508704eada0579"} Jul 10 13:55:27.426222 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:27.426163 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cqp6x" podStartSLOduration=1.060577805 podStartE2EDuration="4.426147017s" totalImagesPullingTime="3.365569212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:55:27.424800567 +0000 UTC m=+334.888406029" watchObservedRunningTime="2026-07-10 13:55:27.426147017 +0000 UTC m=+334.889752462" Jul 10 13:55:28.115722 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:28.115691 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:29.371906 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:29.371866 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:30.321558 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:30.321522 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:38.943844 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:38.943791 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:39.043989 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:39.043959 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:42.269532 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:42.269498 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:42.365325 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:42.365295 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:42.861846 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:42.861797 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:43.108145 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.108112 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:43.209082 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.209050 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:43.246983 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.246956 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk"] Jul 10 13:55:43.250342 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.250320 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:55:43.253098 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.253065 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k9vzt\"" Jul 10 13:55:43.253098 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.253097 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 13:55:43.253297 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.253128 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 13:55:43.253297 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.253067 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 13:55:43.253413 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.253401 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 13:55:43.265670 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.265644 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk"] Jul 10 13:55:43.356073 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.356033 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21d44d5e-e3bd-426a-8fa6-e40557b41a9c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-545b4db9db-g84xk\" (UID: \"21d44d5e-e3bd-426a-8fa6-e40557b41a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:55:43.356073 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.356074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21d44d5e-e3bd-426a-8fa6-e40557b41a9c-manifests\") pod \"opendatahub-operator-controller-manager-545b4db9db-g84xk\" (UID: \"21d44d5e-e3bd-426a-8fa6-e40557b41a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:55:43.356473 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.356129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq9qh\" (UniqueName: \"kubernetes.io/projected/21d44d5e-e3bd-426a-8fa6-e40557b41a9c-kube-api-access-qq9qh\") pod \"opendatahub-operator-controller-manager-545b4db9db-g84xk\" (UID: \"21d44d5e-e3bd-426a-8fa6-e40557b41a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:55:43.356473 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.356175 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21d44d5e-e3bd-426a-8fa6-e40557b41a9c-webhook-cert\") pod \"opendatahub-operator-controller-manager-545b4db9db-g84xk\" (UID: \"21d44d5e-e3bd-426a-8fa6-e40557b41a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:55:43.356473 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.356222 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21d44d5e-e3bd-426a-8fa6-e40557b41a9c-tmp\") pod \"opendatahub-operator-controller-manager-545b4db9db-g84xk\" (UID: \"21d44d5e-e3bd-426a-8fa6-e40557b41a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:55:43.376578 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.376554 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:43.456787 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.456761 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21d44d5e-e3bd-426a-8fa6-e40557b41a9c-tmp\") pod \"opendatahub-operator-controller-manager-545b4db9db-g84xk\" (UID: \"21d44d5e-e3bd-426a-8fa6-e40557b41a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:55:43.456977 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.456795 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21d44d5e-e3bd-426a-8fa6-e40557b41a9c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-545b4db9db-g84xk\" (UID: \"21d44d5e-e3bd-426a-8fa6-e40557b41a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:55:43.456977 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.456841 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21d44d5e-e3bd-426a-8fa6-e40557b41a9c-manifests\") pod \"opendatahub-operator-controller-manager-545b4db9db-g84xk\" (UID: \"21d44d5e-e3bd-426a-8fa6-e40557b41a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:55:43.456977 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.456881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq9qh\" (UniqueName: \"kubernetes.io/projected/21d44d5e-e3bd-426a-8fa6-e40557b41a9c-kube-api-access-qq9qh\") pod \"opendatahub-operator-controller-manager-545b4db9db-g84xk\" (UID: \"21d44d5e-e3bd-426a-8fa6-e40557b41a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:55:43.456977 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.456925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21d44d5e-e3bd-426a-8fa6-e40557b41a9c-webhook-cert\") pod \"opendatahub-operator-controller-manager-545b4db9db-g84xk\" (UID: \"21d44d5e-e3bd-426a-8fa6-e40557b41a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:55:43.457350 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.457294 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21d44d5e-e3bd-426a-8fa6-e40557b41a9c-manifests\") pod \"opendatahub-operator-controller-manager-545b4db9db-g84xk\" (UID: \"21d44d5e-e3bd-426a-8fa6-e40557b41a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:55:43.459346 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.459284 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21d44d5e-e3bd-426a-8fa6-e40557b41a9c-tmp\") pod \"opendatahub-operator-controller-manager-545b4db9db-g84xk\" (UID: \"21d44d5e-e3bd-426a-8fa6-e40557b41a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:55:43.459661 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.459641 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21d44d5e-e3bd-426a-8fa6-e40557b41a9c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-545b4db9db-g84xk\" (UID: \"21d44d5e-e3bd-426a-8fa6-e40557b41a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:55:43.459750 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.459729 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21d44d5e-e3bd-426a-8fa6-e40557b41a9c-webhook-cert\") pod \"opendatahub-operator-controller-manager-545b4db9db-g84xk\" (UID: \"21d44d5e-e3bd-426a-8fa6-e40557b41a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:55:43.467371 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.467351 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq9qh\" (UniqueName: \"kubernetes.io/projected/21d44d5e-e3bd-426a-8fa6-e40557b41a9c-kube-api-access-qq9qh\") pod \"opendatahub-operator-controller-manager-545b4db9db-g84xk\" (UID: \"21d44d5e-e3bd-426a-8fa6-e40557b41a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:55:43.470983 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.470947 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:43.561110 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.561074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:55:43.568357 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.568333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:43.668172 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.668139 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:43.693705 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.693675 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk"] Jul 10 13:55:43.696738 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:55:43.696715 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d44d5e_e3bd_426a_8fa6_e40557b41a9c.slice/crio-47d5d61ce51bbd8ec1e904e46410f896e681d74e2c3b0a1b6c6f492856b8cd7c WatchSource:0}: Error finding container 47d5d61ce51bbd8ec1e904e46410f896e681d74e2c3b0a1b6c6f492856b8cd7c: Status 404 returned error can't find the container with id 47d5d61ce51bbd8ec1e904e46410f896e681d74e2c3b0a1b6c6f492856b8cd7c Jul 10 13:55:43.769049 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:43.768969 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:44.066206 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:44.066117 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:44.163347 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:44.163316 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:44.466236 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:44.466189 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" event={"ID":"21d44d5e-e3bd-426a-8fa6-e40557b41a9c","Type":"ContainerStarted","Data":"47d5d61ce51bbd8ec1e904e46410f896e681d74e2c3b0a1b6c6f492856b8cd7c"} Jul 10 13:55:47.481170 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:47.481132 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" event={"ID":"21d44d5e-e3bd-426a-8fa6-e40557b41a9c","Type":"ContainerStarted","Data":"81fa183350f2584cc2596974dbcf9af5f33401e1c80c8b8035211dc641c6d98f"} Jul 10 13:55:48.486501 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:48.486470 2576 generic.go:356] "Generic (PLEG): container finished" podID="21d44d5e-e3bd-426a-8fa6-e40557b41a9c" containerID="81fa183350f2584cc2596974dbcf9af5f33401e1c80c8b8035211dc641c6d98f" exitCode=0 Jul 10 13:55:48.486905 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:48.486516 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" event={"ID":"21d44d5e-e3bd-426a-8fa6-e40557b41a9c","Type":"ContainerDied","Data":"81fa183350f2584cc2596974dbcf9af5f33401e1c80c8b8035211dc641c6d98f"} Jul 10 13:55:49.492169 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:49.492134 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" event={"ID":"21d44d5e-e3bd-426a-8fa6-e40557b41a9c","Type":"ContainerStarted","Data":"428ef15072373e976e420fd39dbd2e7ded84cee5e936d046052095025e5378a6"} Jul 10 13:55:49.492624 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:49.492229 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:55:49.529599 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:49.529530 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" podStartSLOduration=2.092407561 podStartE2EDuration="6.529509593s" totalImagesPullingTime="3.3461137s" totalInitContainerRuntime="1.090988332s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:55:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:55:49.527117536 +0000 UTC m=+356.990722991" watchObservedRunningTime="2026-07-10 13:55:49.529509593 +0000 UTC m=+356.993115043" Jul 10 13:55:52.122852 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:52.122795 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:52.456799 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:52.456766 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:53.941327 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:53.941295 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:54.037235 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:54.037202 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:54.276153 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:54.276074 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:55:54.347743 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:55:54.347717 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:00.498787 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:00.498757 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-545b4db9db-g84xk" Jul 10 13:56:03.201494 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:03.201465 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:03.330346 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:03.330302 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:03.890599 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:03.890563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:04.054871 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:04.054830 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:04.246775 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:04.246736 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:04.348158 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:04.348082 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:04.453254 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:04.453218 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:04.735615 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:04.735585 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:05.191463 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:05.191426 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6j79w"] Jul 10 13:56:05.201340 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:05.201311 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6j79w" Jul 10 13:56:05.205219 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:05.205193 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 10 13:56:05.206302 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:05.206276 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6j79w"] Jul 10 13:56:05.206736 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:05.206716 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-n7ddz\"" Jul 10 13:56:05.319153 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:05.319122 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:05.338903 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:05.338864 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2tv2\" (UniqueName: \"kubernetes.io/projected/4f1fd144-6cfd-458c-af0a-0139608099a0-kube-api-access-z2tv2\") pod \"model-serving-api-54d89bd5fc-6j79w\" (UID: \"4f1fd144-6cfd-458c-af0a-0139608099a0\") " pod="opendatahub/model-serving-api-54d89bd5fc-6j79w" Jul 10 13:56:05.339092 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:05.338931 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1fd144-6cfd-458c-af0a-0139608099a0-tls-certs\") pod \"model-serving-api-54d89bd5fc-6j79w\" (UID: \"4f1fd144-6cfd-458c-af0a-0139608099a0\") " pod="opendatahub/model-serving-api-54d89bd5fc-6j79w" Jul 10 13:56:05.406261 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:05.406225 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:05.439762 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:05.439719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tv2\" (UniqueName: \"kubernetes.io/projected/4f1fd144-6cfd-458c-af0a-0139608099a0-kube-api-access-z2tv2\") pod \"model-serving-api-54d89bd5fc-6j79w\" (UID: \"4f1fd144-6cfd-458c-af0a-0139608099a0\") " pod="opendatahub/model-serving-api-54d89bd5fc-6j79w" Jul 10 13:56:05.439976 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:05.439789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1fd144-6cfd-458c-af0a-0139608099a0-tls-certs\") pod \"model-serving-api-54d89bd5fc-6j79w\" (UID: \"4f1fd144-6cfd-458c-af0a-0139608099a0\") " pod="opendatahub/model-serving-api-54d89bd5fc-6j79w" Jul 10 13:56:05.442577 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:05.442510 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1fd144-6cfd-458c-af0a-0139608099a0-tls-certs\") pod \"model-serving-api-54d89bd5fc-6j79w\" (UID: \"4f1fd144-6cfd-458c-af0a-0139608099a0\") " pod="opendatahub/model-serving-api-54d89bd5fc-6j79w" Jul 10 13:56:05.448690 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:05.448664 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2tv2\" (UniqueName: \"kubernetes.io/projected/4f1fd144-6cfd-458c-af0a-0139608099a0-kube-api-access-z2tv2\") pod \"model-serving-api-54d89bd5fc-6j79w\" (UID: \"4f1fd144-6cfd-458c-af0a-0139608099a0\") " pod="opendatahub/model-serving-api-54d89bd5fc-6j79w" Jul 10 13:56:05.474827 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:05.474780 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:05.513047 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:05.513005 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6j79w" Jul 10 13:56:05.652060 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:05.651931 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6j79w"] Jul 10 13:56:05.654450 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:56:05.654420 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1fd144_6cfd_458c_af0a_0139608099a0.slice/crio-a13d3b4f30ff0fc4fe86a85262661a3cd43741abfea43935127d54f37dd0f599 WatchSource:0}: Error finding container a13d3b4f30ff0fc4fe86a85262661a3cd43741abfea43935127d54f37dd0f599: Status 404 returned error can't find the container with id a13d3b4f30ff0fc4fe86a85262661a3cd43741abfea43935127d54f37dd0f599 Jul 10 13:56:06.386913 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:06.386852 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:06.564041 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:06.564003 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6j79w" event={"ID":"4f1fd144-6cfd-458c-af0a-0139608099a0","Type":"ContainerStarted","Data":"a13d3b4f30ff0fc4fe86a85262661a3cd43741abfea43935127d54f37dd0f599"} Jul 10 13:56:08.573248 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:08.573217 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6j79w" event={"ID":"4f1fd144-6cfd-458c-af0a-0139608099a0","Type":"ContainerStarted","Data":"650ade20bc355f41361a077a99f4f93403dddc83acae736926412ff8200b832a"} Jul 10 13:56:08.573634 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:08.573277 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-6j79w" Jul 10 13:56:08.593920 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:08.593834 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-6j79w" podStartSLOduration=1.6882937839999999 podStartE2EDuration="3.593798189s" totalImagesPullingTime="1.905504405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:56:08.593325899 +0000 UTC m=+376.056931369" watchObservedRunningTime="2026-07-10 13:56:08.593798189 +0000 UTC m=+376.057403638" Jul 10 13:56:09.259785 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.259749 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:09.360065 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.360025 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:09.513292 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.513198 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj"] Jul 10 13:56:09.516567 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.516537 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.519326 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.519305 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 13:56:09.519510 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.519489 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-9j4dt\"" Jul 10 13:56:09.519594 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.519523 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 13:56:09.519594 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.519524 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 13:56:09.532243 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.529573 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj"] Jul 10 13:56:09.553110 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.547163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:09.614753 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.614712 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6"] Jul 10 13:56:09.617733 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.617715 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.632317 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.632290 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6"] Jul 10 13:56:09.677166 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.677127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceac78c8-3feb-45fa-b098-177137c93b45-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.677357 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.677183 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.677357 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.677212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.677357 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.677237 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.677357 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.677304 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceac78c8-3feb-45fa-b098-177137c93b45-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.677561 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.677390 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.677561 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.677413 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6djk7\" (UniqueName: \"kubernetes.io/projected/ceac78c8-3feb-45fa-b098-177137c93b45-kube-api-access-6djk7\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.677561 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.677487 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceac78c8-3feb-45fa-b098-177137c93b45-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.677561 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.677510 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.777986 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.777897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.777986 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.777947 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceac78c8-3feb-45fa-b098-177137c93b45-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.777986 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.777983 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9slsc\" (UniqueName: \"kubernetes.io/projected/afc01637-a228-4494-8ef8-cc79be42da25-kube-api-access-9slsc\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.778261 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.778261 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.778261 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.778261 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778076 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc01637-a228-4494-8ef8-cc79be42da25-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.778261 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceac78c8-3feb-45fa-b098-177137c93b45-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.778261 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778135 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc01637-a228-4494-8ef8-cc79be42da25-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.778261 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778161 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc01637-a228-4494-8ef8-cc79be42da25-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.778261 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.778261 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6djk7\" (UniqueName: \"kubernetes.io/projected/ceac78c8-3feb-45fa-b098-177137c93b45-kube-api-access-6djk7\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.778261 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778234 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc01637-a228-4494-8ef8-cc79be42da25-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.778261 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc01637-a228-4494-8ef8-cc79be42da25-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.778743 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc01637-a228-4494-8ef8-cc79be42da25-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.778743 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778312 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc01637-a228-4494-8ef8-cc79be42da25-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.778743 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc01637-a228-4494-8ef8-cc79be42da25-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.778743 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778390 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceac78c8-3feb-45fa-b098-177137c93b45-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.778743 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778488 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.778743 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778638 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.778743 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778681 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.779091 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.778844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.779091 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.779057 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceac78c8-3feb-45fa-b098-177137c93b45-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.781105 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.781074 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceac78c8-3feb-45fa-b098-177137c93b45-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.781236 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.781080 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.788052 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.788030 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceac78c8-3feb-45fa-b098-177137c93b45-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.788298 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.788279 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6djk7\" (UniqueName: \"kubernetes.io/projected/ceac78c8-3feb-45fa-b098-177137c93b45-kube-api-access-6djk7\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.835641 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.835607 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:09.851429 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.851397 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:09.879513 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.878801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc01637-a228-4494-8ef8-cc79be42da25-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.879513 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.878870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc01637-a228-4494-8ef8-cc79be42da25-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.879513 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.878908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc01637-a228-4494-8ef8-cc79be42da25-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.879513 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.878931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc01637-a228-4494-8ef8-cc79be42da25-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.879513 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.878962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc01637-a228-4494-8ef8-cc79be42da25-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.879513 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.878998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc01637-a228-4494-8ef8-cc79be42da25-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.879513 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.879085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9slsc\" (UniqueName: \"kubernetes.io/projected/afc01637-a228-4494-8ef8-cc79be42da25-kube-api-access-9slsc\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.879513 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.879127 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc01637-a228-4494-8ef8-cc79be42da25-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.879513 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.879154 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc01637-a228-4494-8ef8-cc79be42da25-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.879513 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.879266 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc01637-a228-4494-8ef8-cc79be42da25-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.879513 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.879342 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc01637-a228-4494-8ef8-cc79be42da25-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.880208 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.879658 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc01637-a228-4494-8ef8-cc79be42da25-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.880208 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.879681 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc01637-a228-4494-8ef8-cc79be42da25-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.880208 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.880005 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc01637-a228-4494-8ef8-cc79be42da25-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.881693 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.881669 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc01637-a228-4494-8ef8-cc79be42da25-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.882141 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.882125 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc01637-a228-4494-8ef8-cc79be42da25-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.887308 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.887288 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc01637-a228-4494-8ef8-cc79be42da25-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.887996 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.887980 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9slsc\" (UniqueName: \"kubernetes.io/projected/afc01637-a228-4494-8ef8-cc79be42da25-kube-api-access-9slsc\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jvlz6\" (UID: \"afc01637-a228-4494-8ef8-cc79be42da25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.928552 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.928498 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:09.952996 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.952962 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:09.992121 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:09.992081 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj"] Jul 10 13:56:09.994474 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:56:09.994441 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceac78c8_3feb_45fa_b098_177137c93b45.slice/crio-ebf722bb8dc7a4dea3572fe462a92902f8bfed2c4bc422c451ee4775bd04f933 WatchSource:0}: Error finding container ebf722bb8dc7a4dea3572fe462a92902f8bfed2c4bc422c451ee4775bd04f933: Status 404 returned error can't find the container with id ebf722bb8dc7a4dea3572fe462a92902f8bfed2c4bc422c451ee4775bd04f933 Jul 10 13:56:10.076398 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:10.076309 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6"] Jul 10 13:56:10.079259 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:56:10.079232 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc01637_a228_4494_8ef8_cc79be42da25.slice/crio-12d5aa63abdf9101429c40119ec51a03b146dffbc2f45840f50b3e3b755c4b3b WatchSource:0}: Error finding container 12d5aa63abdf9101429c40119ec51a03b146dffbc2f45840f50b3e3b755c4b3b: Status 404 returned error can't find the container with id 12d5aa63abdf9101429c40119ec51a03b146dffbc2f45840f50b3e3b755c4b3b Jul 10 13:56:10.584733 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:10.584693 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" event={"ID":"ceac78c8-3feb-45fa-b098-177137c93b45","Type":"ContainerStarted","Data":"ebf722bb8dc7a4dea3572fe462a92902f8bfed2c4bc422c451ee4775bd04f933"} Jul 10 13:56:10.586260 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:10.586220 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" event={"ID":"afc01637-a228-4494-8ef8-cc79be42da25","Type":"ContainerStarted","Data":"12d5aa63abdf9101429c40119ec51a03b146dffbc2f45840f50b3e3b755c4b3b"} Jul 10 13:56:12.033311 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.033281 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk"] Jul 10 13:56:12.036440 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.036417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" Jul 10 13:56:12.040920 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.040874 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 13:56:12.041772 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.041751 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 13:56:12.041922 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.041800 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-m8j48\"" Jul 10 13:56:12.041922 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.041854 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 13:56:12.053607 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.053582 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk"] Jul 10 13:56:12.198495 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.198462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/557d8d62-4252-4155-af82-2c3dbe914f21-metrics-cert\") pod \"lws-controller-manager-745c8d65dc-z65kk\" (UID: \"557d8d62-4252-4155-af82-2c3dbe914f21\") " pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" Jul 10 13:56:12.198674 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.198516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/557d8d62-4252-4155-af82-2c3dbe914f21-manager-config\") pod \"lws-controller-manager-745c8d65dc-z65kk\" (UID: \"557d8d62-4252-4155-af82-2c3dbe914f21\") " pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" Jul 10 13:56:12.198674 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.198539 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/557d8d62-4252-4155-af82-2c3dbe914f21-cert\") pod \"lws-controller-manager-745c8d65dc-z65kk\" (UID: \"557d8d62-4252-4155-af82-2c3dbe914f21\") " pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" Jul 10 13:56:12.198674 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.198592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnmkb\" (UniqueName: \"kubernetes.io/projected/557d8d62-4252-4155-af82-2c3dbe914f21-kube-api-access-hnmkb\") pod \"lws-controller-manager-745c8d65dc-z65kk\" (UID: \"557d8d62-4252-4155-af82-2c3dbe914f21\") " pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" Jul 10 13:56:12.299669 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.299580 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/557d8d62-4252-4155-af82-2c3dbe914f21-manager-config\") pod \"lws-controller-manager-745c8d65dc-z65kk\" (UID: \"557d8d62-4252-4155-af82-2c3dbe914f21\") " pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" Jul 10 13:56:12.299669 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.299620 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/557d8d62-4252-4155-af82-2c3dbe914f21-cert\") pod \"lws-controller-manager-745c8d65dc-z65kk\" (UID: \"557d8d62-4252-4155-af82-2c3dbe914f21\") " pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" Jul 10 13:56:12.299669 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.299663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnmkb\" (UniqueName: \"kubernetes.io/projected/557d8d62-4252-4155-af82-2c3dbe914f21-kube-api-access-hnmkb\") pod \"lws-controller-manager-745c8d65dc-z65kk\" (UID: \"557d8d62-4252-4155-af82-2c3dbe914f21\") " pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" Jul 10 13:56:12.299964 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.299846 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/557d8d62-4252-4155-af82-2c3dbe914f21-metrics-cert\") pod \"lws-controller-manager-745c8d65dc-z65kk\" (UID: \"557d8d62-4252-4155-af82-2c3dbe914f21\") " pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" Jul 10 13:56:12.300347 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.300326 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/557d8d62-4252-4155-af82-2c3dbe914f21-manager-config\") pod \"lws-controller-manager-745c8d65dc-z65kk\" (UID: \"557d8d62-4252-4155-af82-2c3dbe914f21\") " pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" Jul 10 13:56:12.302423 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.302403 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/557d8d62-4252-4155-af82-2c3dbe914f21-cert\") pod \"lws-controller-manager-745c8d65dc-z65kk\" (UID: \"557d8d62-4252-4155-af82-2c3dbe914f21\") " pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" Jul 10 13:56:12.302577 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.302555 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/557d8d62-4252-4155-af82-2c3dbe914f21-metrics-cert\") pod \"lws-controller-manager-745c8d65dc-z65kk\" (UID: \"557d8d62-4252-4155-af82-2c3dbe914f21\") " pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" Jul 10 13:56:12.308673 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.308649 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnmkb\" (UniqueName: \"kubernetes.io/projected/557d8d62-4252-4155-af82-2c3dbe914f21-kube-api-access-hnmkb\") pod \"lws-controller-manager-745c8d65dc-z65kk\" (UID: \"557d8d62-4252-4155-af82-2c3dbe914f21\") " pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" Jul 10 13:56:12.347164 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.347127 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" Jul 10 13:56:12.701032 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.700989 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:56:12.701145 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.701085 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:56:12.701145 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.701134 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:56:12.707504 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.707471 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:56:12.707580 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.707559 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:56:12.707660 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.707642 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:56:12.751314 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:12.751293 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk"] Jul 10 13:56:12.753753 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:56:12.753729 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557d8d62_4252_4155_af82_2c3dbe914f21.slice/crio-8c1027bb20c3b39bebabfb9f15f5370d11ba06798f9dae187d580031491a668a WatchSource:0}: Error finding container 8c1027bb20c3b39bebabfb9f15f5370d11ba06798f9dae187d580031491a668a: Status 404 returned error can't find the container with id 8c1027bb20c3b39bebabfb9f15f5370d11ba06798f9dae187d580031491a668a Jul 10 13:56:13.602690 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:13.602642 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" event={"ID":"afc01637-a228-4494-8ef8-cc79be42da25","Type":"ContainerStarted","Data":"9799a4124e87005b25b2e0256e0087c4d7f805a5beaae1efcc7c6652d4ec3739"} Jul 10 13:56:13.604782 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:13.604740 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" event={"ID":"ceac78c8-3feb-45fa-b098-177137c93b45","Type":"ContainerStarted","Data":"0116b39ff6eecba8bb62d05070f769b85d24c97ba47f51fc699d3d51c059ec81"} Jul 10 13:56:13.606177 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:13.606141 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" event={"ID":"557d8d62-4252-4155-af82-2c3dbe914f21","Type":"ContainerStarted","Data":"8c1027bb20c3b39bebabfb9f15f5370d11ba06798f9dae187d580031491a668a"} Jul 10 13:56:13.653913 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:13.653841 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" podStartSLOduration=2.028180243 podStartE2EDuration="4.65379785s" totalImagesPullingTime="2.625617607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:56:13.651359716 +0000 UTC m=+381.114965159" watchObservedRunningTime="2026-07-10 13:56:13.65379785 +0000 UTC m=+381.117403298" Jul 10 13:56:13.693661 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:13.693605 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" podStartSLOduration=1.989213466 podStartE2EDuration="4.693588468s" totalImagesPullingTime="2.704375002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:56:13.69182555 +0000 UTC m=+381.155431027" watchObservedRunningTime="2026-07-10 13:56:13.693588468 +0000 UTC m=+381.157193914" Jul 10 13:56:13.836785 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:13.836739 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:13.838558 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:13.838505 2576 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body= Jul 10 13:56:13.838760 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:13.838581 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" podUID="ceac78c8-3feb-45fa-b098-177137c93b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" Jul 10 13:56:13.929405 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:13.929361 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:13.934530 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:13.934505 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:14.044319 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:14.044286 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:14.163069 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:14.163038 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:14.346184 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:14.346047 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:14.547594 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:14.547562 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:14.612247 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:14.612139 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" event={"ID":"557d8d62-4252-4155-af82-2c3dbe914f21","Type":"ContainerStarted","Data":"a7dee9240761d261c2ba34255ea745862c10d0a0238f1717be8dcf8093a6bf8f"} Jul 10 13:56:14.612637 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:14.612510 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" Jul 10 13:56:14.612637 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:14.612549 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:14.613383 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:14.613365 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jvlz6" Jul 10 13:56:14.700947 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:14.700866 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" podStartSLOduration=2.03701738 podStartE2EDuration="3.700834147s" totalImagesPullingTime="1.663816767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:56:14.637238974 +0000 UTC m=+382.100844419" watchObservedRunningTime="2026-07-10 13:56:14.700834147 +0000 UTC m=+382.164439594" Jul 10 13:56:14.769284 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:14.769248 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj"] Jul 10 13:56:14.836656 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:14.836619 2576 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body= Jul 10 13:56:14.836851 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:14.836674 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" podUID="ceac78c8-3feb-45fa-b098-177137c93b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" Jul 10 13:56:15.380380 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:15.380341 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:15.477194 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:15.477162 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:15.535856 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:15.535825 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:15.836572 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:15.836537 2576 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body= Jul 10 13:56:15.837031 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:15.836593 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" podUID="ceac78c8-3feb-45fa-b098-177137c93b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" Jul 10 13:56:16.619963 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:16.619917 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" podUID="ceac78c8-3feb-45fa-b098-177137c93b45" containerName="istio-proxy" containerID="cri-o://0116b39ff6eecba8bb62d05070f769b85d24c97ba47f51fc699d3d51c059ec81" gracePeriod=30 Jul 10 13:56:17.357392 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:17.357349 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:19.581933 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:19.581907 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-6j79w" Jul 10 13:56:21.868091 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.868062 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:21.883396 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.883308 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-istio-envoy\") pod \"ceac78c8-3feb-45fa-b098-177137c93b45\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " Jul 10 13:56:21.883396 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.883369 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-credential-socket\") pod \"ceac78c8-3feb-45fa-b098-177137c93b45\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " Jul 10 13:56:21.883396 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.883395 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-workload-certs\") pod \"ceac78c8-3feb-45fa-b098-177137c93b45\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " Jul 10 13:56:21.883668 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.883417 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceac78c8-3feb-45fa-b098-177137c93b45-istio-podinfo\") pod \"ceac78c8-3feb-45fa-b098-177137c93b45\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " Jul 10 13:56:21.883668 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.883447 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-istio-data\") pod \"ceac78c8-3feb-45fa-b098-177137c93b45\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " Jul 10 13:56:21.883668 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.883486 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6djk7\" (UniqueName: \"kubernetes.io/projected/ceac78c8-3feb-45fa-b098-177137c93b45-kube-api-access-6djk7\") pod \"ceac78c8-3feb-45fa-b098-177137c93b45\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " Jul 10 13:56:21.883668 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.883512 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceac78c8-3feb-45fa-b098-177137c93b45-istio-token\") pod \"ceac78c8-3feb-45fa-b098-177137c93b45\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " Jul 10 13:56:21.883668 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.883541 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceac78c8-3feb-45fa-b098-177137c93b45-istiod-ca-cert\") pod \"ceac78c8-3feb-45fa-b098-177137c93b45\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " Jul 10 13:56:21.883668 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.883594 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-workload-socket\") pod \"ceac78c8-3feb-45fa-b098-177137c93b45\" (UID: \"ceac78c8-3feb-45fa-b098-177137c93b45\") " Jul 10 13:56:21.883969 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.883681 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ceac78c8-3feb-45fa-b098-177137c93b45" (UID: "ceac78c8-3feb-45fa-b098-177137c93b45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:56:21.883969 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.883739 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ceac78c8-3feb-45fa-b098-177137c93b45" (UID: "ceac78c8-3feb-45fa-b098-177137c93b45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:56:21.883969 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.883762 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-istio-data" (OuterVolumeSpecName: "istio-data") pod "ceac78c8-3feb-45fa-b098-177137c93b45" (UID: "ceac78c8-3feb-45fa-b098-177137c93b45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:56:21.883969 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.883906 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-credential-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 13:56:21.883969 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.883927 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-workload-certs\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 13:56:21.883969 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.883929 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ceac78c8-3feb-45fa-b098-177137c93b45" (UID: "ceac78c8-3feb-45fa-b098-177137c93b45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:56:21.883969 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.883941 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-istio-data\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 13:56:21.884277 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.884109 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceac78c8-3feb-45fa-b098-177137c93b45-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ceac78c8-3feb-45fa-b098-177137c93b45" (UID: "ceac78c8-3feb-45fa-b098-177137c93b45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:56:21.885741 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.885713 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ceac78c8-3feb-45fa-b098-177137c93b45-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ceac78c8-3feb-45fa-b098-177137c93b45" (UID: "ceac78c8-3feb-45fa-b098-177137c93b45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 13:56:21.885868 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.885751 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceac78c8-3feb-45fa-b098-177137c93b45-kube-api-access-6djk7" (OuterVolumeSpecName: "kube-api-access-6djk7") pod "ceac78c8-3feb-45fa-b098-177137c93b45" (UID: "ceac78c8-3feb-45fa-b098-177137c93b45"). InnerVolumeSpecName "kube-api-access-6djk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:56:21.885868 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.885777 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ceac78c8-3feb-45fa-b098-177137c93b45" (UID: "ceac78c8-3feb-45fa-b098-177137c93b45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:56:21.885868 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.885838 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceac78c8-3feb-45fa-b098-177137c93b45-istio-token" (OuterVolumeSpecName: "istio-token") pod "ceac78c8-3feb-45fa-b098-177137c93b45" (UID: "ceac78c8-3feb-45fa-b098-177137c93b45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:56:21.985091 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.985050 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceac78c8-3feb-45fa-b098-177137c93b45-istio-podinfo\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 13:56:21.985091 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.985083 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6djk7\" (UniqueName: \"kubernetes.io/projected/ceac78c8-3feb-45fa-b098-177137c93b45-kube-api-access-6djk7\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 13:56:21.985091 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.985092 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceac78c8-3feb-45fa-b098-177137c93b45-istio-token\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 13:56:21.985091 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.985101 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceac78c8-3feb-45fa-b098-177137c93b45-istiod-ca-cert\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 13:56:21.985354 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.985109 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-workload-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 13:56:21.985354 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:21.985116 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceac78c8-3feb-45fa-b098-177137c93b45-istio-envoy\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 13:56:22.645674 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:22.645642 2576 generic.go:356] "Generic (PLEG): container finished" podID="ceac78c8-3feb-45fa-b098-177137c93b45" containerID="0116b39ff6eecba8bb62d05070f769b85d24c97ba47f51fc699d3d51c059ec81" exitCode=0 Jul 10 13:56:22.645865 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:22.645688 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" event={"ID":"ceac78c8-3feb-45fa-b098-177137c93b45","Type":"ContainerDied","Data":"0116b39ff6eecba8bb62d05070f769b85d24c97ba47f51fc699d3d51c059ec81"} Jul 10 13:56:22.645865 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:22.645694 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" Jul 10 13:56:22.645865 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:22.645729 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj" event={"ID":"ceac78c8-3feb-45fa-b098-177137c93b45","Type":"ContainerDied","Data":"ebf722bb8dc7a4dea3572fe462a92902f8bfed2c4bc422c451ee4775bd04f933"} Jul 10 13:56:22.645865 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:22.645750 2576 scope.go:117] "RemoveContainer" containerID="0116b39ff6eecba8bb62d05070f769b85d24c97ba47f51fc699d3d51c059ec81" Jul 10 13:56:22.654252 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:22.654236 2576 scope.go:117] "RemoveContainer" containerID="0116b39ff6eecba8bb62d05070f769b85d24c97ba47f51fc699d3d51c059ec81" Jul 10 13:56:22.654517 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:56:22.654488 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0116b39ff6eecba8bb62d05070f769b85d24c97ba47f51fc699d3d51c059ec81\": container with ID starting with 0116b39ff6eecba8bb62d05070f769b85d24c97ba47f51fc699d3d51c059ec81 not found: ID does not exist" containerID="0116b39ff6eecba8bb62d05070f769b85d24c97ba47f51fc699d3d51c059ec81" Jul 10 13:56:22.654563 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:22.654527 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0116b39ff6eecba8bb62d05070f769b85d24c97ba47f51fc699d3d51c059ec81"} err="failed to get container status \"0116b39ff6eecba8bb62d05070f769b85d24c97ba47f51fc699d3d51c059ec81\": rpc error: code = NotFound desc = could not find container \"0116b39ff6eecba8bb62d05070f769b85d24c97ba47f51fc699d3d51c059ec81\": container with ID starting with 0116b39ff6eecba8bb62d05070f769b85d24c97ba47f51fc699d3d51c059ec81 not found: ID does not exist" Jul 10 13:56:22.667869 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:22.667841 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj"] Jul 10 13:56:22.672361 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:22.672337 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-d8dsj"] Jul 10 13:56:23.035870 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:23.035787 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceac78c8-3feb-45fa-b098-177137c93b45" path="/var/lib/kubelet/pods/ceac78c8-3feb-45fa-b098-177137c93b45/volumes" Jul 10 13:56:26.623887 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:26.623851 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-745c8d65dc-z65kk" Jul 10 13:56:50.149604 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:50.149569 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:56:55.270104 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:56:55.270068 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:04.600463 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:04.600427 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:04.708054 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:04.708022 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:04.840996 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:04.840964 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:05.563047 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:05.563018 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:05.771133 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:05.771097 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:10.005545 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:10.005513 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:19.900385 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:19.900269 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:20.785824 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:20.785778 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:20.881100 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:20.881066 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:20.981573 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:20.981539 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:33.922223 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:33.922188 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:37.820651 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:37.820608 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:40.221723 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:40.221685 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:40.430612 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:40.430576 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:40.791182 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:40.791150 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:41.078014 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:41.077928 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:41.174821 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:41.174778 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:42.873827 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:42.873790 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:44.028091 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:44.028054 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:44.175310 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:44.175277 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:46.069719 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:46.069681 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:49.221453 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:49.221420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:49.335723 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:49.335693 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:52.623615 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:52.623582 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:52.683588 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:52.683557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:57.771422 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.771387 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-swbkn"] Jul 10 13:57:57.771886 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.771792 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceac78c8-3feb-45fa-b098-177137c93b45" containerName="istio-proxy" Jul 10 13:57:57.771886 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.771820 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceac78c8-3feb-45fa-b098-177137c93b45" containerName="istio-proxy" Jul 10 13:57:57.771971 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.771904 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceac78c8-3feb-45fa-b098-177137c93b45" containerName="istio-proxy" Jul 10 13:57:57.775076 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.775053 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-swbkn" Jul 10 13:57:57.777381 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.777350 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:57.777881 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.777863 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 13:57:57.778470 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.778457 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 13:57:57.778533 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.778459 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 13:57:57.778533 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.778459 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 13:57:57.779049 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.779033 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tpptj\"" Jul 10 13:57:57.784269 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.784248 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-swbkn"] Jul 10 13:57:57.867824 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.867783 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a1078386-e19c-4f8e-bf73-36ba6db1cadb-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-swbkn\" (UID: \"a1078386-e19c-4f8e-bf73-36ba6db1cadb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-swbkn" Jul 10 13:57:57.867994 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.867833 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stmr4\" (UniqueName: \"kubernetes.io/projected/a1078386-e19c-4f8e-bf73-36ba6db1cadb-kube-api-access-stmr4\") pod \"kuadrant-console-plugin-66b5774985-swbkn\" (UID: \"a1078386-e19c-4f8e-bf73-36ba6db1cadb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-swbkn" Jul 10 13:57:57.867994 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.867904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1078386-e19c-4f8e-bf73-36ba6db1cadb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-swbkn\" (UID: \"a1078386-e19c-4f8e-bf73-36ba6db1cadb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-swbkn" Jul 10 13:57:57.968794 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.968762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stmr4\" (UniqueName: \"kubernetes.io/projected/a1078386-e19c-4f8e-bf73-36ba6db1cadb-kube-api-access-stmr4\") pod \"kuadrant-console-plugin-66b5774985-swbkn\" (UID: \"a1078386-e19c-4f8e-bf73-36ba6db1cadb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-swbkn" Jul 10 13:57:57.968998 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.968831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1078386-e19c-4f8e-bf73-36ba6db1cadb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-swbkn\" (UID: \"a1078386-e19c-4f8e-bf73-36ba6db1cadb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-swbkn" Jul 10 13:57:57.968998 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.968907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a1078386-e19c-4f8e-bf73-36ba6db1cadb-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-swbkn\" (UID: \"a1078386-e19c-4f8e-bf73-36ba6db1cadb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-swbkn" Jul 10 13:57:57.968998 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:57:57.968934 2576 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 10 13:57:57.969121 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:57:57.969008 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1078386-e19c-4f8e-bf73-36ba6db1cadb-plugin-serving-cert podName:a1078386-e19c-4f8e-bf73-36ba6db1cadb nodeName:}" failed. No retries permitted until 2026-07-10 13:57:58.468989769 +0000 UTC m=+485.932595197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/a1078386-e19c-4f8e-bf73-36ba6db1cadb-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-swbkn" (UID: "a1078386-e19c-4f8e-bf73-36ba6db1cadb") : secret "plugin-serving-cert" not found Jul 10 13:57:57.969516 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.969499 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a1078386-e19c-4f8e-bf73-36ba6db1cadb-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-swbkn\" (UID: \"a1078386-e19c-4f8e-bf73-36ba6db1cadb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-swbkn" Jul 10 13:57:57.983898 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:57.983872 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stmr4\" (UniqueName: \"kubernetes.io/projected/a1078386-e19c-4f8e-bf73-36ba6db1cadb-kube-api-access-stmr4\") pod \"kuadrant-console-plugin-66b5774985-swbkn\" (UID: \"a1078386-e19c-4f8e-bf73-36ba6db1cadb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-swbkn" Jul 10 13:57:58.182123 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:58.182089 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:58.418305 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:58.418253 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:58.474388 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:58.474292 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1078386-e19c-4f8e-bf73-36ba6db1cadb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-swbkn\" (UID: \"a1078386-e19c-4f8e-bf73-36ba6db1cadb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-swbkn" Jul 10 13:57:58.476672 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:58.476651 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1078386-e19c-4f8e-bf73-36ba6db1cadb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-swbkn\" (UID: \"a1078386-e19c-4f8e-bf73-36ba6db1cadb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-swbkn" Jul 10 13:57:58.522008 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:58.521974 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:58.684763 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:58.684732 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:58.685641 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:58.685625 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-swbkn" Jul 10 13:57:58.771960 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:58.771927 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:57:58.834725 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:58.834699 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-swbkn"] Jul 10 13:57:58.837180 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:57:58.837150 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1078386_e19c_4f8e_bf73_36ba6db1cadb.slice/crio-092ae08b872495eec2f54681a4a1378f9aae6126dcfd8ecde8ecf75691f1dd4c WatchSource:0}: Error finding container 092ae08b872495eec2f54681a4a1378f9aae6126dcfd8ecde8ecf75691f1dd4c: Status 404 returned error can't find the container with id 092ae08b872495eec2f54681a4a1378f9aae6126dcfd8ecde8ecf75691f1dd4c Jul 10 13:57:59.036054 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:57:59.035972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-swbkn" event={"ID":"a1078386-e19c-4f8e-bf73-36ba6db1cadb","Type":"ContainerStarted","Data":"092ae08b872495eec2f54681a4a1378f9aae6126dcfd8ecde8ecf75691f1dd4c"} Jul 10 13:58:01.756114 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:01.756056 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:01.860770 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:01.860701 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:07.855004 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:07.854966 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc"] Jul 10 13:58:07.875189 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:07.875157 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc"] Jul 10 13:58:07.875365 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:07.875272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc" Jul 10 13:58:07.878556 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:07.878532 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-68m97\"" Jul 10 13:58:07.960332 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:07.960290 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ee20074-4963-43d5-8c2e-c42c782aa2e5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dmcqc\" (UID: \"8ee20074-4963-43d5-8c2e-c42c782aa2e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc" Jul 10 13:58:07.960508 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:07.960341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpvs7\" (UniqueName: \"kubernetes.io/projected/8ee20074-4963-43d5-8c2e-c42c782aa2e5-kube-api-access-hpvs7\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dmcqc\" (UID: \"8ee20074-4963-43d5-8c2e-c42c782aa2e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc" Jul 10 13:58:08.061798 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.061758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ee20074-4963-43d5-8c2e-c42c782aa2e5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dmcqc\" (UID: \"8ee20074-4963-43d5-8c2e-c42c782aa2e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc" Jul 10 13:58:08.061798 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.061819 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpvs7\" (UniqueName: \"kubernetes.io/projected/8ee20074-4963-43d5-8c2e-c42c782aa2e5-kube-api-access-hpvs7\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dmcqc\" (UID: \"8ee20074-4963-43d5-8c2e-c42c782aa2e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc" Jul 10 13:58:08.062226 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.062199 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ee20074-4963-43d5-8c2e-c42c782aa2e5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dmcqc\" (UID: \"8ee20074-4963-43d5-8c2e-c42c782aa2e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc" Jul 10 13:58:08.071902 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.071879 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpvs7\" (UniqueName: \"kubernetes.io/projected/8ee20074-4963-43d5-8c2e-c42c782aa2e5-kube-api-access-hpvs7\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dmcqc\" (UID: \"8ee20074-4963-43d5-8c2e-c42c782aa2e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc" Jul 10 13:58:08.188519 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.188480 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc" Jul 10 13:58:08.402958 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.402927 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:08.504365 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.504273 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:08.613803 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.613766 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc"] Jul 10 13:58:08.620683 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.620656 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc"] Jul 10 13:58:08.639243 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.639214 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22"] Jul 10 13:58:08.650473 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.650436 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22" Jul 10 13:58:08.670079 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.670052 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22"] Jul 10 13:58:08.770574 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.770493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqs5f\" (UniqueName: \"kubernetes.io/projected/7ad22878-307f-41a7-b633-ecd9d37198b7-kube-api-access-fqs5f\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qxk22\" (UID: \"7ad22878-307f-41a7-b633-ecd9d37198b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22" Jul 10 13:58:08.770574 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.770570 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ad22878-307f-41a7-b633-ecd9d37198b7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qxk22\" (UID: \"7ad22878-307f-41a7-b633-ecd9d37198b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22" Jul 10 13:58:08.872056 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.872012 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqs5f\" (UniqueName: \"kubernetes.io/projected/7ad22878-307f-41a7-b633-ecd9d37198b7-kube-api-access-fqs5f\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qxk22\" (UID: \"7ad22878-307f-41a7-b633-ecd9d37198b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22" Jul 10 13:58:08.872512 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.872096 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ad22878-307f-41a7-b633-ecd9d37198b7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qxk22\" (UID: \"7ad22878-307f-41a7-b633-ecd9d37198b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22" Jul 10 13:58:08.872564 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.872544 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ad22878-307f-41a7-b633-ecd9d37198b7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qxk22\" (UID: \"7ad22878-307f-41a7-b633-ecd9d37198b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22" Jul 10 13:58:08.886417 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.886383 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqs5f\" (UniqueName: \"kubernetes.io/projected/7ad22878-307f-41a7-b633-ecd9d37198b7-kube-api-access-fqs5f\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qxk22\" (UID: \"7ad22878-307f-41a7-b633-ecd9d37198b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22" Jul 10 13:58:08.964087 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:08.964046 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22" Jul 10 13:58:11.398718 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:11.398682 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:11.494592 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:11.494547 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:16.182993 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:16.182955 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:16.242899 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:16.242864 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:16.384769 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:16.384682 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:16.483998 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:16.483916 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:20.315293 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:20.315269 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22"] Jul 10 13:58:20.317736 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:58:20.317705 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad22878_307f_41a7_b633_ecd9d37198b7.slice/crio-cad3dbad2f6127e8f8520b8393a97a71515fd67365f09d9e2d9fcea4a38efdc8 WatchSource:0}: Error finding container cad3dbad2f6127e8f8520b8393a97a71515fd67365f09d9e2d9fcea4a38efdc8: Status 404 returned error can't find the container with id cad3dbad2f6127e8f8520b8393a97a71515fd67365f09d9e2d9fcea4a38efdc8 Jul 10 13:58:21.145800 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:21.145763 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22" event={"ID":"7ad22878-307f-41a7-b633-ecd9d37198b7","Type":"ContainerStarted","Data":"cad3dbad2f6127e8f8520b8393a97a71515fd67365f09d9e2d9fcea4a38efdc8"} Jul 10 13:58:21.147265 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:21.147242 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-swbkn" event={"ID":"a1078386-e19c-4f8e-bf73-36ba6db1cadb","Type":"ContainerStarted","Data":"e12bc3dddb9c6d3cf67181e5a0b1abd2b9509a9944f9417756ac947b2627f065"} Jul 10 13:58:21.178691 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:21.178643 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-swbkn" podStartSLOduration=2.774536842 podStartE2EDuration="24.178626774s" totalImagesPullingTime="21.404089932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:58:21.176742992 +0000 UTC m=+508.640348439" watchObservedRunningTime="2026-07-10 13:58:21.178626774 +0000 UTC m=+508.642232219" Jul 10 13:58:21.420032 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:21.419931 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:21.535054 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:21.535018 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:22.702020 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:58:22.701963 2576 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-dmcqc_kuadrant-system_8ee20074-4963-43d5-8c2e-c42c782aa2e5_0(2a91154b5390a255054e3e3ad59d4ae8f60563448feecbe6928e6196a6f7abe4): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-dmcqc to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 10 13:58:22.702364 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:58:22.702052 2576 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-dmcqc_kuadrant-system_8ee20074-4963-43d5-8c2e-c42c782aa2e5_0(2a91154b5390a255054e3e3ad59d4ae8f60563448feecbe6928e6196a6f7abe4): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-dmcqc to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc" Jul 10 13:58:23.156343 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:23.156317 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc" Jul 10 13:58:23.158961 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:58:23.158930 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-dmcqc\" is forbidden: User \"system:node:ip-10-0-143-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-25.ec2.internal' and this object" podUID="8ee20074-4963-43d5-8c2e-c42c782aa2e5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc" Jul 10 13:58:23.163908 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:23.163890 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc" Jul 10 13:58:23.166526 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:58:23.166500 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-dmcqc\" is forbidden: User \"system:node:ip-10-0-143-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-25.ec2.internal' and this object" podUID="8ee20074-4963-43d5-8c2e-c42c782aa2e5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc" Jul 10 13:58:23.305480 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:23.305444 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpvs7\" (UniqueName: \"kubernetes.io/projected/8ee20074-4963-43d5-8c2e-c42c782aa2e5-kube-api-access-hpvs7\") pod \"8ee20074-4963-43d5-8c2e-c42c782aa2e5\" (UID: \"8ee20074-4963-43d5-8c2e-c42c782aa2e5\") " Jul 10 13:58:23.305640 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:23.305536 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ee20074-4963-43d5-8c2e-c42c782aa2e5-extensions-socket-volume\") pod \"8ee20074-4963-43d5-8c2e-c42c782aa2e5\" (UID: \"8ee20074-4963-43d5-8c2e-c42c782aa2e5\") " Jul 10 13:58:23.305840 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:23.305800 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee20074-4963-43d5-8c2e-c42c782aa2e5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8ee20074-4963-43d5-8c2e-c42c782aa2e5" (UID: "8ee20074-4963-43d5-8c2e-c42c782aa2e5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:58:23.307554 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:23.307529 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee20074-4963-43d5-8c2e-c42c782aa2e5-kube-api-access-hpvs7" (OuterVolumeSpecName: "kube-api-access-hpvs7") pod "8ee20074-4963-43d5-8c2e-c42c782aa2e5" (UID: "8ee20074-4963-43d5-8c2e-c42c782aa2e5"). InnerVolumeSpecName "kube-api-access-hpvs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:58:23.406099 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:23.406066 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ee20074-4963-43d5-8c2e-c42c782aa2e5-extensions-socket-volume\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 13:58:23.406099 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:23.406092 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpvs7\" (UniqueName: \"kubernetes.io/projected/8ee20074-4963-43d5-8c2e-c42c782aa2e5-kube-api-access-hpvs7\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 13:58:24.161671 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:24.161638 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc" Jul 10 13:58:24.164234 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:58:24.164197 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-dmcqc\" is forbidden: User \"system:node:ip-10-0-143-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-25.ec2.internal' and this object" podUID="8ee20074-4963-43d5-8c2e-c42c782aa2e5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc" Jul 10 13:58:24.172238 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:58:24.172195 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-dmcqc\" is forbidden: User \"system:node:ip-10-0-143-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-25.ec2.internal' and this object" podUID="8ee20074-4963-43d5-8c2e-c42c782aa2e5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dmcqc" Jul 10 13:58:25.037676 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:25.037635 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ee20074-4963-43d5-8c2e-c42c782aa2e5" path="/var/lib/kubelet/pods/8ee20074-4963-43d5-8c2e-c42c782aa2e5/volumes" Jul 10 13:58:26.171875 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:26.171843 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22" event={"ID":"7ad22878-307f-41a7-b633-ecd9d37198b7","Type":"ContainerStarted","Data":"e761da669d45907f298b81625da04a9d5c8cbee31cf9ee86e4c6bdf40e338612"} Jul 10 13:58:26.172251 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:26.171915 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22" Jul 10 13:58:26.195832 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:26.195773 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22" podStartSLOduration=12.834346445 podStartE2EDuration="18.19575906s" totalImagesPullingTime="5.361412615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:58:26.194553108 +0000 UTC m=+513.658158555" watchObservedRunningTime="2026-07-10 13:58:26.19575906 +0000 UTC m=+513.659364557" Jul 10 13:58:36.348983 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:36.348951 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:36.446938 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:36.446905 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:37.178779 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:37.178754 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22" Jul 10 13:58:48.153804 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:48.153770 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:49.169555 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:49.169523 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22"] Jul 10 13:58:49.170009 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:49.169710 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22" podUID="7ad22878-307f-41a7-b633-ecd9d37198b7" containerName="manager" containerID="cri-o://e761da669d45907f298b81625da04a9d5c8cbee31cf9ee86e4c6bdf40e338612" gracePeriod=10 Jul 10 13:58:49.402562 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:49.402537 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22" Jul 10 13:58:49.523374 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:49.523288 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ad22878-307f-41a7-b633-ecd9d37198b7-extensions-socket-volume\") pod \"7ad22878-307f-41a7-b633-ecd9d37198b7\" (UID: \"7ad22878-307f-41a7-b633-ecd9d37198b7\") " Jul 10 13:58:49.523374 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:49.523332 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqs5f\" (UniqueName: \"kubernetes.io/projected/7ad22878-307f-41a7-b633-ecd9d37198b7-kube-api-access-fqs5f\") pod \"7ad22878-307f-41a7-b633-ecd9d37198b7\" (UID: \"7ad22878-307f-41a7-b633-ecd9d37198b7\") " Jul 10 13:58:49.523914 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:49.523887 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad22878-307f-41a7-b633-ecd9d37198b7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7ad22878-307f-41a7-b633-ecd9d37198b7" (UID: "7ad22878-307f-41a7-b633-ecd9d37198b7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:58:49.525476 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:49.525458 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad22878-307f-41a7-b633-ecd9d37198b7-kube-api-access-fqs5f" (OuterVolumeSpecName: "kube-api-access-fqs5f") pod "7ad22878-307f-41a7-b633-ecd9d37198b7" (UID: "7ad22878-307f-41a7-b633-ecd9d37198b7"). InnerVolumeSpecName "kube-api-access-fqs5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:58:49.624167 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:49.624130 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ad22878-307f-41a7-b633-ecd9d37198b7-extensions-socket-volume\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 13:58:49.624167 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:49.624160 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqs5f\" (UniqueName: \"kubernetes.io/projected/7ad22878-307f-41a7-b633-ecd9d37198b7-kube-api-access-fqs5f\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 13:58:50.273495 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:50.273459 2576 generic.go:356] "Generic (PLEG): container finished" podID="7ad22878-307f-41a7-b633-ecd9d37198b7" containerID="e761da669d45907f298b81625da04a9d5c8cbee31cf9ee86e4c6bdf40e338612" exitCode=0 Jul 10 13:58:50.273919 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:50.273506 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22" Jul 10 13:58:50.273919 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:50.273519 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22" event={"ID":"7ad22878-307f-41a7-b633-ecd9d37198b7","Type":"ContainerDied","Data":"e761da669d45907f298b81625da04a9d5c8cbee31cf9ee86e4c6bdf40e338612"} Jul 10 13:58:50.273919 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:50.273563 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22" event={"ID":"7ad22878-307f-41a7-b633-ecd9d37198b7","Type":"ContainerDied","Data":"cad3dbad2f6127e8f8520b8393a97a71515fd67365f09d9e2d9fcea4a38efdc8"} Jul 10 13:58:50.273919 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:50.273584 2576 scope.go:117] "RemoveContainer" containerID="e761da669d45907f298b81625da04a9d5c8cbee31cf9ee86e4c6bdf40e338612" Jul 10 13:58:50.282997 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:50.282977 2576 scope.go:117] "RemoveContainer" containerID="e761da669d45907f298b81625da04a9d5c8cbee31cf9ee86e4c6bdf40e338612" Jul 10 13:58:50.283296 ip-10-0-143-25 kubenswrapper[2576]: E0710 13:58:50.283277 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e761da669d45907f298b81625da04a9d5c8cbee31cf9ee86e4c6bdf40e338612\": container with ID starting with e761da669d45907f298b81625da04a9d5c8cbee31cf9ee86e4c6bdf40e338612 not found: ID does not exist" containerID="e761da669d45907f298b81625da04a9d5c8cbee31cf9ee86e4c6bdf40e338612" Jul 10 13:58:50.283340 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:50.283306 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e761da669d45907f298b81625da04a9d5c8cbee31cf9ee86e4c6bdf40e338612"} err="failed to get container status \"e761da669d45907f298b81625da04a9d5c8cbee31cf9ee86e4c6bdf40e338612\": rpc error: code = NotFound desc = could not find container \"e761da669d45907f298b81625da04a9d5c8cbee31cf9ee86e4c6bdf40e338612\": container with ID starting with e761da669d45907f298b81625da04a9d5c8cbee31cf9ee86e4c6bdf40e338612 not found: ID does not exist" Jul 10 13:58:50.298350 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:50.298321 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22"] Jul 10 13:58:50.302405 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:50.302383 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qxk22"] Jul 10 13:58:51.035796 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:51.035761 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ad22878-307f-41a7-b633-ecd9d37198b7" path="/var/lib/kubelet/pods/7ad22878-307f-41a7-b633-ecd9d37198b7/volumes" Jul 10 13:58:53.603490 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.603457 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln"] Jul 10 13:58:53.603879 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.603776 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad22878-307f-41a7-b633-ecd9d37198b7" containerName="manager" Jul 10 13:58:53.603879 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.603788 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad22878-307f-41a7-b633-ecd9d37198b7" containerName="manager" Jul 10 13:58:53.603879 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.603863 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad22878-307f-41a7-b633-ecd9d37198b7" containerName="manager" Jul 10 13:58:53.630485 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.630454 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln"] Jul 10 13:58:53.630643 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.630557 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.632139 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.632110 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:53.633232 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.633213 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xvrwb\"" Jul 10 13:58:53.761130 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.761099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad868c2-3722-4d72-883c-1e1780c6ddce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.761300 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.761136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad868c2-3722-4d72-883c-1e1780c6ddce-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.761300 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.761156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad868c2-3722-4d72-883c-1e1780c6ddce-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.761300 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.761176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad868c2-3722-4d72-883c-1e1780c6ddce-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.761300 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.761264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad868c2-3722-4d72-883c-1e1780c6ddce-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.761509 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.761310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzms6\" (UniqueName: \"kubernetes.io/projected/6ad868c2-3722-4d72-883c-1e1780c6ddce-kube-api-access-bzms6\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.761509 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.761358 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad868c2-3722-4d72-883c-1e1780c6ddce-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.761509 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.761435 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad868c2-3722-4d72-883c-1e1780c6ddce-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.761509 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.761461 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad868c2-3722-4d72-883c-1e1780c6ddce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.862287 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.862202 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzms6\" (UniqueName: \"kubernetes.io/projected/6ad868c2-3722-4d72-883c-1e1780c6ddce-kube-api-access-bzms6\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.862287 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.862253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad868c2-3722-4d72-883c-1e1780c6ddce-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.862508 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.862291 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad868c2-3722-4d72-883c-1e1780c6ddce-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.862508 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.862313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad868c2-3722-4d72-883c-1e1780c6ddce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.862508 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.862344 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad868c2-3722-4d72-883c-1e1780c6ddce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.862508 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.862489 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad868c2-3722-4d72-883c-1e1780c6ddce-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.862716 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.862537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad868c2-3722-4d72-883c-1e1780c6ddce-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.862716 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.862573 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad868c2-3722-4d72-883c-1e1780c6ddce-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.862716 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.862625 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad868c2-3722-4d72-883c-1e1780c6ddce-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.862908 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.862727 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad868c2-3722-4d72-883c-1e1780c6ddce-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.862908 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.862889 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad868c2-3722-4d72-883c-1e1780c6ddce-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.863013 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.862957 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad868c2-3722-4d72-883c-1e1780c6ddce-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.863185 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.863164 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad868c2-3722-4d72-883c-1e1780c6ddce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.863241 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.863178 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad868c2-3722-4d72-883c-1e1780c6ddce-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.864542 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.864520 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad868c2-3722-4d72-883c-1e1780c6ddce-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.864912 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.864893 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad868c2-3722-4d72-883c-1e1780c6ddce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.870233 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.870212 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad868c2-3722-4d72-883c-1e1780c6ddce-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.870631 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.870611 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzms6\" (UniqueName: \"kubernetes.io/projected/6ad868c2-3722-4d72-883c-1e1780c6ddce-kube-api-access-bzms6\") pod \"maas-default-gateway-openshift-default-56654dcf7-8tjln\" (UID: \"6ad868c2-3722-4d72-883c-1e1780c6ddce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:53.874219 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.874191 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:53.941794 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:53.941759 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:54.088067 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:54.088036 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln"] Jul 10 13:58:54.091747 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:58:54.091721 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad868c2_3722_4d72_883c_1e1780c6ddce.slice/crio-810fa10253dbbab0e4a08806b05cc163a9f14c9426db68df8978cfb7dbcfb65c WatchSource:0}: Error finding container 810fa10253dbbab0e4a08806b05cc163a9f14c9426db68df8978cfb7dbcfb65c: Status 404 returned error can't find the container with id 810fa10253dbbab0e4a08806b05cc163a9f14c9426db68df8978cfb7dbcfb65c Jul 10 13:58:54.093926 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:54.093886 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:58:54.094019 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:54.093962 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:58:54.094019 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:54.094000 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:58:54.293518 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:54.293486 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" event={"ID":"6ad868c2-3722-4d72-883c-1e1780c6ddce","Type":"ContainerStarted","Data":"f0c4d6b81d863101c3ce493d84fe289c4b3d8c3c17951ed057dcb436d5dcdc6b"} Jul 10 13:58:54.293678 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:54.293545 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" event={"ID":"6ad868c2-3722-4d72-883c-1e1780c6ddce","Type":"ContainerStarted","Data":"810fa10253dbbab0e4a08806b05cc163a9f14c9426db68df8978cfb7dbcfb65c"} Jul 10 13:58:54.314097 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:54.314052 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" podStartSLOduration=1.3140375469999999 podStartE2EDuration="1.314037547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:58:54.31249113 +0000 UTC m=+541.776096580" watchObservedRunningTime="2026-07-10 13:58:54.314037547 +0000 UTC m=+541.777643046" Jul 10 13:58:54.942717 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:54.942687 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:55.947100 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:55.947068 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:56.301377 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:56.301295 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:56.302380 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:56.302356 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 13:58:58.043197 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:58.043167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:58.132750 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:58.132717 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:58.234132 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:58.234092 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:58.332031 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:58.331946 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 13:58:58.335346 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:58.335328 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-hc6r7" Jul 10 13:58:58.338044 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:58.338023 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 13:58:58.346029 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:58.346007 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 13:58:58.378923 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:58.378893 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 13:58:58.403557 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:58.403521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67p9v\" (UniqueName: \"kubernetes.io/projected/0e9d79f9-68a9-484a-bbf2-1ca5b5228d09-kube-api-access-67p9v\") pod \"limitador-limitador-54d6d988d5-hc6r7\" (UID: \"0e9d79f9-68a9-484a-bbf2-1ca5b5228d09\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hc6r7" Jul 10 13:58:58.403717 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:58.403565 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0e9d79f9-68a9-484a-bbf2-1ca5b5228d09-config-file\") pod \"limitador-limitador-54d6d988d5-hc6r7\" (UID: \"0e9d79f9-68a9-484a-bbf2-1ca5b5228d09\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hc6r7" Jul 10 13:58:58.505054 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:58.505012 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67p9v\" (UniqueName: \"kubernetes.io/projected/0e9d79f9-68a9-484a-bbf2-1ca5b5228d09-kube-api-access-67p9v\") pod \"limitador-limitador-54d6d988d5-hc6r7\" (UID: \"0e9d79f9-68a9-484a-bbf2-1ca5b5228d09\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hc6r7" Jul 10 13:58:58.505215 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:58.505065 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0e9d79f9-68a9-484a-bbf2-1ca5b5228d09-config-file\") pod \"limitador-limitador-54d6d988d5-hc6r7\" (UID: \"0e9d79f9-68a9-484a-bbf2-1ca5b5228d09\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hc6r7" Jul 10 13:58:58.505654 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:58.505636 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0e9d79f9-68a9-484a-bbf2-1ca5b5228d09-config-file\") pod \"limitador-limitador-54d6d988d5-hc6r7\" (UID: \"0e9d79f9-68a9-484a-bbf2-1ca5b5228d09\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hc6r7" Jul 10 13:58:58.515662 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:58.515635 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67p9v\" (UniqueName: \"kubernetes.io/projected/0e9d79f9-68a9-484a-bbf2-1ca5b5228d09-kube-api-access-67p9v\") pod \"limitador-limitador-54d6d988d5-hc6r7\" (UID: \"0e9d79f9-68a9-484a-bbf2-1ca5b5228d09\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hc6r7" Jul 10 13:58:58.646537 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:58.646510 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-hc6r7" Jul 10 13:58:58.772552 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:58.772440 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 13:58:58.775178 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:58:58.775152 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9d79f9_68a9_484a_bbf2_1ca5b5228d09.slice/crio-560d69231654f06710cb54ae607853155cd6f74f3b6c0fb638f144076b8066f4 WatchSource:0}: Error finding container 560d69231654f06710cb54ae607853155cd6f74f3b6c0fb638f144076b8066f4: Status 404 returned error can't find the container with id 560d69231654f06710cb54ae607853155cd6f74f3b6c0fb638f144076b8066f4 Jul 10 13:58:58.893989 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:58.893957 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:58.976779 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:58.976692 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:58:59.315260 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:58:59.315185 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-hc6r7" event={"ID":"0e9d79f9-68a9-484a-bbf2-1ca5b5228d09","Type":"ContainerStarted","Data":"560d69231654f06710cb54ae607853155cd6f74f3b6c0fb638f144076b8066f4"} Jul 10 13:59:02.329972 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:02.329934 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-hc6r7" event={"ID":"0e9d79f9-68a9-484a-bbf2-1ca5b5228d09","Type":"ContainerStarted","Data":"f696d193aa646d2daab9e2bf1f8cc8f584c2a611bb7ba2438d9027729cb51abe"} Jul 10 13:59:02.330343 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:02.330023 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-hc6r7" Jul 10 13:59:02.353483 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:02.353425 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-hc6r7" podStartSLOduration=1.622123335 podStartE2EDuration="4.353409845s" totalImagesPullingTime="2.73128651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:59:02.352933724 +0000 UTC m=+549.816539172" watchObservedRunningTime="2026-07-10 13:59:02.353409845 +0000 UTC m=+549.817015291" Jul 10 13:59:13.334724 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:13.334697 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-hc6r7" Jul 10 13:59:16.729069 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:16.729029 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:59:16.817592 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:16.817561 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:59:17.313955 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:17.313917 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:59:29.827534 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:29.827493 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:59:30.278979 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:30.278943 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:59:30.668121 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:30.668084 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f4d44ffb6-2hzk5"] Jul 10 13:59:30.671136 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:30.671119 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f4d44ffb6-2hzk5" Jul 10 13:59:30.675114 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:30.675095 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hh8xv\"" Jul 10 13:59:30.675114 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:30.675094 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 10 13:59:30.686023 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:30.685998 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f4d44ffb6-2hzk5"] Jul 10 13:59:30.781680 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:30.781643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdsll\" (UniqueName: \"kubernetes.io/projected/72a775fe-bcff-4b86-a56e-664631c2a21b-kube-api-access-mdsll\") pod \"authorino-f4d44ffb6-2hzk5\" (UID: \"72a775fe-bcff-4b86-a56e-664631c2a21b\") " pod="kuadrant-system/authorino-f4d44ffb6-2hzk5" Jul 10 13:59:30.781680 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:30.781697 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/72a775fe-bcff-4b86-a56e-664631c2a21b-tls-cert\") pod \"authorino-f4d44ffb6-2hzk5\" (UID: \"72a775fe-bcff-4b86-a56e-664631c2a21b\") " pod="kuadrant-system/authorino-f4d44ffb6-2hzk5" Jul 10 13:59:30.882587 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:30.882549 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdsll\" (UniqueName: \"kubernetes.io/projected/72a775fe-bcff-4b86-a56e-664631c2a21b-kube-api-access-mdsll\") pod \"authorino-f4d44ffb6-2hzk5\" (UID: \"72a775fe-bcff-4b86-a56e-664631c2a21b\") " pod="kuadrant-system/authorino-f4d44ffb6-2hzk5" Jul 10 13:59:30.883022 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:30.882597 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/72a775fe-bcff-4b86-a56e-664631c2a21b-tls-cert\") pod \"authorino-f4d44ffb6-2hzk5\" (UID: \"72a775fe-bcff-4b86-a56e-664631c2a21b\") " pod="kuadrant-system/authorino-f4d44ffb6-2hzk5" Jul 10 13:59:30.884990 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:30.884970 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/72a775fe-bcff-4b86-a56e-664631c2a21b-tls-cert\") pod \"authorino-f4d44ffb6-2hzk5\" (UID: \"72a775fe-bcff-4b86-a56e-664631c2a21b\") " pod="kuadrant-system/authorino-f4d44ffb6-2hzk5" Jul 10 13:59:30.896502 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:30.896468 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdsll\" (UniqueName: \"kubernetes.io/projected/72a775fe-bcff-4b86-a56e-664631c2a21b-kube-api-access-mdsll\") pod \"authorino-f4d44ffb6-2hzk5\" (UID: \"72a775fe-bcff-4b86-a56e-664631c2a21b\") " pod="kuadrant-system/authorino-f4d44ffb6-2hzk5" Jul 10 13:59:30.980003 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:30.979915 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f4d44ffb6-2hzk5" Jul 10 13:59:31.126625 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:31.126537 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f4d44ffb6-2hzk5"] Jul 10 13:59:31.129310 ip-10-0-143-25 kubenswrapper[2576]: W0710 13:59:31.129283 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72a775fe_bcff_4b86_a56e_664631c2a21b.slice/crio-5c70b9c8de25159bb18ffbe7909f40f765f26fc1eadef2e387bf2c73f6402923 WatchSource:0}: Error finding container 5c70b9c8de25159bb18ffbe7909f40f765f26fc1eadef2e387bf2c73f6402923: Status 404 returned error can't find the container with id 5c70b9c8de25159bb18ffbe7909f40f765f26fc1eadef2e387bf2c73f6402923 Jul 10 13:59:31.455000 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:31.454966 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f4d44ffb6-2hzk5" event={"ID":"72a775fe-bcff-4b86-a56e-664631c2a21b","Type":"ContainerStarted","Data":"5c70b9c8de25159bb18ffbe7909f40f765f26fc1eadef2e387bf2c73f6402923"} Jul 10 13:59:34.470874 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:34.470775 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f4d44ffb6-2hzk5" event={"ID":"72a775fe-bcff-4b86-a56e-664631c2a21b","Type":"ContainerStarted","Data":"d34cf63e263ce404c848e3daa76e87b42191857d9591e391105608a82b67be2b"} Jul 10 13:59:34.493878 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:34.493796 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-f4d44ffb6-2hzk5" podStartSLOduration=1.451043066 podStartE2EDuration="4.493780586s" totalImagesPullingTime="3.04273752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:59:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:59:34.493694661 +0000 UTC m=+581.957300110" watchObservedRunningTime="2026-07-10 13:59:34.493780586 +0000 UTC m=+581.957386033" Jul 10 13:59:39.378514 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:39.378478 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:59:39.468607 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:39.468574 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:59:39.569674 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:39.569642 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:59:42.159754 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:42.159720 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:59:42.261153 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:42.261119 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:59:46.182830 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:46.182764 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:59:46.285722 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:46.285693 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 13:59:52.958986 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:52.958960 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 13:59:52.959550 ip-10-0-143-25 kubenswrapper[2576]: I0710 13:59:52.959526 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 14:00:15.524932 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:15.524788 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:00:15.616109 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:15.616077 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:00:15.995976 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:15.995939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:00:16.307084 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:16.306998 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:00:16.675235 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:16.675200 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:00:16.774955 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:16.774925 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:00:16.897666 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:16.897635 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:00:25.694770 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:25.694736 2576 generic.go:356] "Generic (PLEG): container finished" podID="6ad868c2-3722-4d72-883c-1e1780c6ddce" containerID="f0c4d6b81d863101c3ce493d84fe289c4b3d8c3c17951ed057dcb436d5dcdc6b" exitCode=0 Jul 10 14:00:25.695194 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:25.694785 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" event={"ID":"6ad868c2-3722-4d72-883c-1e1780c6ddce","Type":"ContainerDied","Data":"f0c4d6b81d863101c3ce493d84fe289c4b3d8c3c17951ed057dcb436d5dcdc6b"} Jul 10 14:00:25.695194 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:25.695174 2576 scope.go:117] "RemoveContainer" containerID="f0c4d6b81d863101c3ce493d84fe289c4b3d8c3c17951ed057dcb436d5dcdc6b" Jul 10 14:00:25.696290 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:25.696253 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:00:25.696399 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:25.696347 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:00:25.696497 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:25.696400 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:00:25.751977 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:25.751949 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:00:25.830651 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:25.830613 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:00:26.302233 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:26.302198 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 14:00:26.700628 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:26.700593 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" event={"ID":"6ad868c2-3722-4d72-883c-1e1780c6ddce","Type":"ContainerStarted","Data":"b442d1922e23a8d817c0ede1130b4124c1f4c8ab364da10a43257c3f74388611"} Jul 10 14:00:26.943355 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:26.943316 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 14:00:26.943564 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:26.943442 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-8tjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 10 14:00:26.943564 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:26.943486 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" podUID="6ad868c2-3722-4d72-883c-1e1780c6ddce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 10 14:00:27.942738 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:27.942695 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-8tjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 10 14:00:27.943822 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:27.943340 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" podUID="6ad868c2-3722-4d72-883c-1e1780c6ddce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 10 14:00:28.943285 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:28.943247 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-8tjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 10 14:00:28.943922 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:28.943886 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" podUID="6ad868c2-3722-4d72-883c-1e1780c6ddce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 10 14:00:29.942583 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:29.942543 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-8tjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 10 14:00:29.942897 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:29.942869 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" podUID="6ad868c2-3722-4d72-883c-1e1780c6ddce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 10 14:00:30.946707 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:30.946677 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 14:00:30.947190 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:30.947017 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 14:00:30.947687 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:30.947669 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8tjln" Jul 10 14:00:47.086823 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:47.086774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:00:47.507629 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:47.507594 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:00:58.138395 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.138363 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk"] Jul 10 14:00:58.140755 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.140736 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.147459 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.147436 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 10 14:00:58.148136 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.148119 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 14:00:58.148220 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.148180 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2k477\"" Jul 10 14:00:58.150241 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.150226 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 14:00:58.159130 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.159102 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk"] Jul 10 14:00:58.220077 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.220048 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:00:58.329990 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.329947 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50bdcb80-0e45-47dc-b78f-846cbf652c4b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r7vzk\" (UID: \"50bdcb80-0e45-47dc-b78f-846cbf652c4b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.330171 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.329999 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50bdcb80-0e45-47dc-b78f-846cbf652c4b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r7vzk\" (UID: \"50bdcb80-0e45-47dc-b78f-846cbf652c4b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.330171 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.330049 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50bdcb80-0e45-47dc-b78f-846cbf652c4b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r7vzk\" (UID: \"50bdcb80-0e45-47dc-b78f-846cbf652c4b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.330171 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.330071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50bdcb80-0e45-47dc-b78f-846cbf652c4b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r7vzk\" (UID: \"50bdcb80-0e45-47dc-b78f-846cbf652c4b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.330171 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.330092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50bdcb80-0e45-47dc-b78f-846cbf652c4b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r7vzk\" (UID: \"50bdcb80-0e45-47dc-b78f-846cbf652c4b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.330171 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.330137 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj4pl\" (UniqueName: \"kubernetes.io/projected/50bdcb80-0e45-47dc-b78f-846cbf652c4b-kube-api-access-nj4pl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r7vzk\" (UID: \"50bdcb80-0e45-47dc-b78f-846cbf652c4b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.431003 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.430965 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50bdcb80-0e45-47dc-b78f-846cbf652c4b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r7vzk\" (UID: \"50bdcb80-0e45-47dc-b78f-846cbf652c4b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.431003 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.431006 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50bdcb80-0e45-47dc-b78f-846cbf652c4b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r7vzk\" (UID: \"50bdcb80-0e45-47dc-b78f-846cbf652c4b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.431228 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.431045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj4pl\" (UniqueName: \"kubernetes.io/projected/50bdcb80-0e45-47dc-b78f-846cbf652c4b-kube-api-access-nj4pl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r7vzk\" (UID: \"50bdcb80-0e45-47dc-b78f-846cbf652c4b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.431228 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.431091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50bdcb80-0e45-47dc-b78f-846cbf652c4b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r7vzk\" (UID: \"50bdcb80-0e45-47dc-b78f-846cbf652c4b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.431228 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.431122 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50bdcb80-0e45-47dc-b78f-846cbf652c4b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r7vzk\" (UID: \"50bdcb80-0e45-47dc-b78f-846cbf652c4b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.431228 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.431155 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50bdcb80-0e45-47dc-b78f-846cbf652c4b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r7vzk\" (UID: \"50bdcb80-0e45-47dc-b78f-846cbf652c4b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.431452 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.431425 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50bdcb80-0e45-47dc-b78f-846cbf652c4b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r7vzk\" (UID: \"50bdcb80-0e45-47dc-b78f-846cbf652c4b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.431510 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.431445 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50bdcb80-0e45-47dc-b78f-846cbf652c4b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r7vzk\" (UID: \"50bdcb80-0e45-47dc-b78f-846cbf652c4b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.431582 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.431562 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50bdcb80-0e45-47dc-b78f-846cbf652c4b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r7vzk\" (UID: \"50bdcb80-0e45-47dc-b78f-846cbf652c4b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.433387 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.433357 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50bdcb80-0e45-47dc-b78f-846cbf652c4b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r7vzk\" (UID: \"50bdcb80-0e45-47dc-b78f-846cbf652c4b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.433508 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.433462 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50bdcb80-0e45-47dc-b78f-846cbf652c4b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r7vzk\" (UID: \"50bdcb80-0e45-47dc-b78f-846cbf652c4b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.442104 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.442077 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj4pl\" (UniqueName: \"kubernetes.io/projected/50bdcb80-0e45-47dc-b78f-846cbf652c4b-kube-api-access-nj4pl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r7vzk\" (UID: \"50bdcb80-0e45-47dc-b78f-846cbf652c4b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.450759 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.450732 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:00:58.589609 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.589587 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk"] Jul 10 14:00:58.591961 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:00:58.591935 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50bdcb80_0e45_47dc_b78f_846cbf652c4b.slice/crio-b50cd35db4e788a1a57d8aeee538f6134edf4aabd7e4b7c031f3e353a4292b90 WatchSource:0}: Error finding container b50cd35db4e788a1a57d8aeee538f6134edf4aabd7e4b7c031f3e353a4292b90: Status 404 returned error can't find the container with id b50cd35db4e788a1a57d8aeee538f6134edf4aabd7e4b7c031f3e353a4292b90 Jul 10 14:00:58.593373 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.593358 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 14:00:58.842157 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.842063 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" event={"ID":"50bdcb80-0e45-47dc-b78f-846cbf652c4b","Type":"ContainerStarted","Data":"b50cd35db4e788a1a57d8aeee538f6134edf4aabd7e4b7c031f3e353a4292b90"} Jul 10 14:00:58.912135 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:00:58.912100 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:01:01.361848 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:01.361796 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:01:06.767274 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:06.767234 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:01:08.008433 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:08.008395 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:01:14.925115 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:14.925064 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" event={"ID":"50bdcb80-0e45-47dc-b78f-846cbf652c4b","Type":"ContainerStarted","Data":"96c18dfcbc72bca929125ad34b2f386e51805a7cf207cdde986853ec0896ec12"} Jul 10 14:01:14.948740 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:14.948633 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" podStartSLOduration=0.844662374 podStartE2EDuration="16.948619351s" totalImagesPullingTime="16.103956977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:01:14.948185609 +0000 UTC m=+682.411791081" watchObservedRunningTime="2026-07-10 14:01:14.948619351 +0000 UTC m=+682.412224796" Jul 10 14:01:18.451742 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:18.451690 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:01:18.452193 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:18.451765 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:01:18.464615 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:18.464587 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:01:18.954549 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:18.954523 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r7vzk" Jul 10 14:01:28.789785 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:28.789746 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:01:30.298353 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:30.298315 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:01:40.291385 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.291350 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw"] Jul 10 14:01:40.294780 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.294763 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.297466 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.297443 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 10 14:01:40.304728 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.304705 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw"] Jul 10 14:01:40.336986 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.336953 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:01:40.372785 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.372748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31377e37-f8eb-4e20-87ef-22ac212289a0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5h9jw\" (UID: \"31377e37-f8eb-4e20-87ef-22ac212289a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.372785 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.372788 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31377e37-f8eb-4e20-87ef-22ac212289a0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5h9jw\" (UID: \"31377e37-f8eb-4e20-87ef-22ac212289a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.373031 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.372834 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31377e37-f8eb-4e20-87ef-22ac212289a0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5h9jw\" (UID: \"31377e37-f8eb-4e20-87ef-22ac212289a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.373031 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.372936 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31377e37-f8eb-4e20-87ef-22ac212289a0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5h9jw\" (UID: \"31377e37-f8eb-4e20-87ef-22ac212289a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.373031 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.372995 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjdgw\" (UniqueName: \"kubernetes.io/projected/31377e37-f8eb-4e20-87ef-22ac212289a0-kube-api-access-tjdgw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5h9jw\" (UID: \"31377e37-f8eb-4e20-87ef-22ac212289a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.373171 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.373055 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31377e37-f8eb-4e20-87ef-22ac212289a0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5h9jw\" (UID: \"31377e37-f8eb-4e20-87ef-22ac212289a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.473543 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.473506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjdgw\" (UniqueName: \"kubernetes.io/projected/31377e37-f8eb-4e20-87ef-22ac212289a0-kube-api-access-tjdgw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5h9jw\" (UID: \"31377e37-f8eb-4e20-87ef-22ac212289a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.473741 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.473557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31377e37-f8eb-4e20-87ef-22ac212289a0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5h9jw\" (UID: \"31377e37-f8eb-4e20-87ef-22ac212289a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.473741 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.473596 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31377e37-f8eb-4e20-87ef-22ac212289a0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5h9jw\" (UID: \"31377e37-f8eb-4e20-87ef-22ac212289a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.473741 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.473617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31377e37-f8eb-4e20-87ef-22ac212289a0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5h9jw\" (UID: \"31377e37-f8eb-4e20-87ef-22ac212289a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.473741 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.473637 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31377e37-f8eb-4e20-87ef-22ac212289a0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5h9jw\" (UID: \"31377e37-f8eb-4e20-87ef-22ac212289a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.473741 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.473674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31377e37-f8eb-4e20-87ef-22ac212289a0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5h9jw\" (UID: \"31377e37-f8eb-4e20-87ef-22ac212289a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.474172 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.474146 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31377e37-f8eb-4e20-87ef-22ac212289a0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5h9jw\" (UID: \"31377e37-f8eb-4e20-87ef-22ac212289a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.474265 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.474183 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31377e37-f8eb-4e20-87ef-22ac212289a0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5h9jw\" (UID: \"31377e37-f8eb-4e20-87ef-22ac212289a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.474265 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.474153 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31377e37-f8eb-4e20-87ef-22ac212289a0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5h9jw\" (UID: \"31377e37-f8eb-4e20-87ef-22ac212289a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.475990 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.475970 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31377e37-f8eb-4e20-87ef-22ac212289a0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5h9jw\" (UID: \"31377e37-f8eb-4e20-87ef-22ac212289a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.476265 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.476244 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31377e37-f8eb-4e20-87ef-22ac212289a0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5h9jw\" (UID: \"31377e37-f8eb-4e20-87ef-22ac212289a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.483088 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.483064 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjdgw\" (UniqueName: \"kubernetes.io/projected/31377e37-f8eb-4e20-87ef-22ac212289a0-kube-api-access-tjdgw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5h9jw\" (UID: \"31377e37-f8eb-4e20-87ef-22ac212289a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.605856 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.605761 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:40.738369 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:40.738182 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw"] Jul 10 14:01:40.741155 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:01:40.741114 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31377e37_f8eb_4e20_87ef_22ac212289a0.slice/crio-ab931f9cebc5aa70845f99c03116db1a4a0e0eb02713190437b0e8bba13b29dd WatchSource:0}: Error finding container ab931f9cebc5aa70845f99c03116db1a4a0e0eb02713190437b0e8bba13b29dd: Status 404 returned error can't find the container with id ab931f9cebc5aa70845f99c03116db1a4a0e0eb02713190437b0e8bba13b29dd Jul 10 14:01:41.039651 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:41.039614 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" event={"ID":"31377e37-f8eb-4e20-87ef-22ac212289a0","Type":"ContainerStarted","Data":"d04d8e33f3fae930b3f2feefa15f5cbe6a1655f000b06f5477ac32e10ec69f80"} Jul 10 14:01:41.039830 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:41.039677 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" event={"ID":"31377e37-f8eb-4e20-87ef-22ac212289a0","Type":"ContainerStarted","Data":"ab931f9cebc5aa70845f99c03116db1a4a0e0eb02713190437b0e8bba13b29dd"} Jul 10 14:01:41.059029 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:41.058973 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" podStartSLOduration=0.865966728 podStartE2EDuration="1.058957722s" totalImagesPullingTime="192.990994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:01:41.057834696 +0000 UTC m=+708.521440144" watchObservedRunningTime="2026-07-10 14:01:41.058957722 +0000 UTC m=+708.522563168" Jul 10 14:01:42.202321 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:42.202287 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:01:44.204323 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:44.204288 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:01:50.606185 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:50.606142 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:50.606660 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:50.606212 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:50.618746 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:50.618714 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:51.098354 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:51.098319 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5h9jw" Jul 10 14:01:53.095501 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:01:53.095469 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:02:36.503879 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:36.503820 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c4f766d87-5vjgs"] Jul 10 14:02:36.506924 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:36.506905 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c4f766d87-5vjgs" Jul 10 14:02:36.515657 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:36.515630 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c4f766d87-5vjgs"] Jul 10 14:02:36.656188 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:36.651633 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/47da23a7-449a-4ce6-a77d-2c77eaaa3060-tls-cert\") pod \"authorino-6c4f766d87-5vjgs\" (UID: \"47da23a7-449a-4ce6-a77d-2c77eaaa3060\") " pod="kuadrant-system/authorino-6c4f766d87-5vjgs" Jul 10 14:02:36.656188 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:36.651728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k5xs\" (UniqueName: \"kubernetes.io/projected/47da23a7-449a-4ce6-a77d-2c77eaaa3060-kube-api-access-8k5xs\") pod \"authorino-6c4f766d87-5vjgs\" (UID: \"47da23a7-449a-4ce6-a77d-2c77eaaa3060\") " pod="kuadrant-system/authorino-6c4f766d87-5vjgs" Jul 10 14:02:36.753041 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:36.752983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/47da23a7-449a-4ce6-a77d-2c77eaaa3060-tls-cert\") pod \"authorino-6c4f766d87-5vjgs\" (UID: \"47da23a7-449a-4ce6-a77d-2c77eaaa3060\") " pod="kuadrant-system/authorino-6c4f766d87-5vjgs" Jul 10 14:02:36.753240 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:36.753074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k5xs\" (UniqueName: \"kubernetes.io/projected/47da23a7-449a-4ce6-a77d-2c77eaaa3060-kube-api-access-8k5xs\") pod \"authorino-6c4f766d87-5vjgs\" (UID: \"47da23a7-449a-4ce6-a77d-2c77eaaa3060\") " pod="kuadrant-system/authorino-6c4f766d87-5vjgs" Jul 10 14:02:36.755661 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:36.755589 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/47da23a7-449a-4ce6-a77d-2c77eaaa3060-tls-cert\") pod \"authorino-6c4f766d87-5vjgs\" (UID: \"47da23a7-449a-4ce6-a77d-2c77eaaa3060\") " pod="kuadrant-system/authorino-6c4f766d87-5vjgs" Jul 10 14:02:36.761650 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:36.761609 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k5xs\" (UniqueName: \"kubernetes.io/projected/47da23a7-449a-4ce6-a77d-2c77eaaa3060-kube-api-access-8k5xs\") pod \"authorino-6c4f766d87-5vjgs\" (UID: \"47da23a7-449a-4ce6-a77d-2c77eaaa3060\") " pod="kuadrant-system/authorino-6c4f766d87-5vjgs" Jul 10 14:02:36.815676 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:36.815638 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c4f766d87-5vjgs" Jul 10 14:02:36.910757 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:36.910721 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:02:36.953372 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:36.953337 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c4f766d87-5vjgs"] Jul 10 14:02:36.956831 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:02:36.956774 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47da23a7_449a_4ce6_a77d_2c77eaaa3060.slice/crio-a35ca2417109b43820923e5e8fdea1fea84d682daa99db0d9f2cc41992195b56 WatchSource:0}: Error finding container a35ca2417109b43820923e5e8fdea1fea84d682daa99db0d9f2cc41992195b56: Status 404 returned error can't find the container with id a35ca2417109b43820923e5e8fdea1fea84d682daa99db0d9f2cc41992195b56 Jul 10 14:02:37.018633 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:37.018539 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:02:37.296458 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:37.296367 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c4f766d87-5vjgs" event={"ID":"47da23a7-449a-4ce6-a77d-2c77eaaa3060","Type":"ContainerStarted","Data":"a35ca2417109b43820923e5e8fdea1fea84d682daa99db0d9f2cc41992195b56"} Jul 10 14:02:38.038327 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:38.038297 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:02:38.137757 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:38.137711 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:02:41.315614 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:41.315579 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c4f766d87-5vjgs" event={"ID":"47da23a7-449a-4ce6-a77d-2c77eaaa3060","Type":"ContainerStarted","Data":"02e570d8411003cb5ede488812a4a48dd30aed285cb25413b9848b792ad32372"} Jul 10 14:02:41.345684 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:41.345616 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6c4f766d87-5vjgs" podStartSLOduration=1.794598446 podStartE2EDuration="5.345592589s" totalImagesPullingTime="3.550994143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:02:41.343882649 +0000 UTC m=+768.807488096" watchObservedRunningTime="2026-07-10 14:02:41.345592589 +0000 UTC m=+768.809198037" Jul 10 14:02:41.413370 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:41.413315 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f4d44ffb6-2hzk5"] Jul 10 14:02:41.413573 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:41.413526 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-f4d44ffb6-2hzk5" podUID="72a775fe-bcff-4b86-a56e-664631c2a21b" containerName="authorino" containerID="cri-o://d34cf63e263ce404c848e3daa76e87b42191857d9591e391105608a82b67be2b" gracePeriod=30 Jul 10 14:02:41.672311 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:41.672281 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f4d44ffb6-2hzk5" Jul 10 14:02:41.797834 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:41.797767 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdsll\" (UniqueName: \"kubernetes.io/projected/72a775fe-bcff-4b86-a56e-664631c2a21b-kube-api-access-mdsll\") pod \"72a775fe-bcff-4b86-a56e-664631c2a21b\" (UID: \"72a775fe-bcff-4b86-a56e-664631c2a21b\") " Jul 10 14:02:41.797834 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:41.797837 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/72a775fe-bcff-4b86-a56e-664631c2a21b-tls-cert\") pod \"72a775fe-bcff-4b86-a56e-664631c2a21b\" (UID: \"72a775fe-bcff-4b86-a56e-664631c2a21b\") " Jul 10 14:02:41.799908 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:41.799869 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a775fe-bcff-4b86-a56e-664631c2a21b-kube-api-access-mdsll" (OuterVolumeSpecName: "kube-api-access-mdsll") pod "72a775fe-bcff-4b86-a56e-664631c2a21b" (UID: "72a775fe-bcff-4b86-a56e-664631c2a21b"). InnerVolumeSpecName "kube-api-access-mdsll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:02:41.808302 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:41.808267 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a775fe-bcff-4b86-a56e-664631c2a21b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "72a775fe-bcff-4b86-a56e-664631c2a21b" (UID: "72a775fe-bcff-4b86-a56e-664631c2a21b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 14:02:41.899213 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:41.899172 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdsll\" (UniqueName: \"kubernetes.io/projected/72a775fe-bcff-4b86-a56e-664631c2a21b-kube-api-access-mdsll\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:02:41.899364 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:41.899214 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/72a775fe-bcff-4b86-a56e-664631c2a21b-tls-cert\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:02:42.321622 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:42.321517 2576 generic.go:356] "Generic (PLEG): container finished" podID="72a775fe-bcff-4b86-a56e-664631c2a21b" containerID="d34cf63e263ce404c848e3daa76e87b42191857d9591e391105608a82b67be2b" exitCode=0 Jul 10 14:02:42.321622 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:42.321558 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f4d44ffb6-2hzk5" Jul 10 14:02:42.321622 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:42.321603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f4d44ffb6-2hzk5" event={"ID":"72a775fe-bcff-4b86-a56e-664631c2a21b","Type":"ContainerDied","Data":"d34cf63e263ce404c848e3daa76e87b42191857d9591e391105608a82b67be2b"} Jul 10 14:02:42.322222 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:42.321647 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f4d44ffb6-2hzk5" event={"ID":"72a775fe-bcff-4b86-a56e-664631c2a21b","Type":"ContainerDied","Data":"5c70b9c8de25159bb18ffbe7909f40f765f26fc1eadef2e387bf2c73f6402923"} Jul 10 14:02:42.322222 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:42.321665 2576 scope.go:117] "RemoveContainer" containerID="d34cf63e263ce404c848e3daa76e87b42191857d9591e391105608a82b67be2b" Jul 10 14:02:42.331665 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:42.331151 2576 scope.go:117] "RemoveContainer" containerID="d34cf63e263ce404c848e3daa76e87b42191857d9591e391105608a82b67be2b" Jul 10 14:02:42.331665 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:02:42.331486 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34cf63e263ce404c848e3daa76e87b42191857d9591e391105608a82b67be2b\": container with ID starting with d34cf63e263ce404c848e3daa76e87b42191857d9591e391105608a82b67be2b not found: ID does not exist" containerID="d34cf63e263ce404c848e3daa76e87b42191857d9591e391105608a82b67be2b" Jul 10 14:02:42.331665 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:42.331517 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34cf63e263ce404c848e3daa76e87b42191857d9591e391105608a82b67be2b"} err="failed to get container status \"d34cf63e263ce404c848e3daa76e87b42191857d9591e391105608a82b67be2b\": rpc error: code = NotFound desc = could not find container \"d34cf63e263ce404c848e3daa76e87b42191857d9591e391105608a82b67be2b\": container with ID starting with d34cf63e263ce404c848e3daa76e87b42191857d9591e391105608a82b67be2b not found: ID does not exist" Jul 10 14:02:42.373479 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:42.373442 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f4d44ffb6-2hzk5"] Jul 10 14:02:42.377678 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:42.377651 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-f4d44ffb6-2hzk5"] Jul 10 14:02:43.035876 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:02:43.035846 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72a775fe-bcff-4b86-a56e-664631c2a21b" path="/var/lib/kubelet/pods/72a775fe-bcff-4b86-a56e-664631c2a21b/volumes" Jul 10 14:03:08.002385 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:03:08.002347 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:03:18.396501 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:03:18.396419 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:03:28.799803 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:03:28.799764 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:03:38.907354 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:03:38.907311 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:03:48.752945 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:03:48.752908 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:03:51.640777 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:03:51.640743 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:03:53.673891 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:03:53.673855 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:04:01.767033 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:04:01.766993 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:04:17.108447 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:04:17.108405 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:04:29.406762 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:04:29.406725 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:04:32.873957 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:04:32.873857 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:04:34.577832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:04:34.577770 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:04:34.672796 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:04:34.672766 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:04:34.796962 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:04:34.796926 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:04:41.046241 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:04:41.046208 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:04:41.118902 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:04:41.118866 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:04:41.214327 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:04:41.214296 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:04:41.339098 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:04:41.339013 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:04:41.448042 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:04:41.448013 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:04:47.548464 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:04:47.548385 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:04:49.820118 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:04:49.820084 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:04:52.986289 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:04:52.986264 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 14:04:52.986720 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:04:52.986264 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 14:05:04.407322 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:05:04.407282 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:05:14.099030 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:05:14.098986 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:05:16.406724 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:05:16.406688 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:05:23.051632 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:05:23.051596 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:05:34.096356 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:05:34.096316 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:05:44.403892 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:05:44.403856 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:05:49.841975 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:05:49.841936 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:05:59.501513 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:05:59.501469 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:06:06.099197 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:06:06.099160 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:06:13.791057 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:06:13.791015 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:06:36.704063 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:06:36.703965 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:06:45.698218 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:06:45.698180 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:06:53.197291 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:06:53.197258 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:07:08.500939 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:07:08.500901 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:07:21.220989 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:07:21.220954 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:07:37.505008 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:07:37.504965 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:07:47.282125 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:07:47.282031 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:07:56.797554 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:07:56.797513 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:08:06.312577 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:08:06.312540 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:08:15.696436 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:08:15.696399 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:09:13.023594 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:09:13.023558 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:09:53.017372 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:09:53.017340 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 14:09:53.018892 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:09:53.018870 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 14:10:03.608630 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:10:03.608590 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:10:12.401919 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:10:12.401886 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:10:25.310140 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:10:25.310101 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:10:36.499210 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:10:36.499176 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:10:46.273791 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:10:46.273695 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:10:46.347948 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:10:46.347910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:10:48.504218 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:10:48.504178 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:10:57.895949 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:10:57.895913 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:11:08.192008 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:11:08.191970 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:11:17.696255 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:11:17.696215 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:11:29.413024 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:11:29.412990 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:11:37.110143 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:11:37.110099 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:11:47.302290 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:11:47.302248 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:11:54.007840 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:11:54.007790 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:12:15.298431 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:12:15.298346 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:12:24.116236 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:12:24.116199 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:12:35.716092 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:12:35.716056 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:12:47.635833 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:12:47.635781 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:13:04.409184 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:13:04.409142 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:13:14.602735 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:13:14.602699 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:13:23.921039 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:13:23.921001 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:13:32.096953 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:13:32.096912 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:13:42.145846 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:13:42.145788 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:13:50.908194 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:13:50.908157 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:13:53.988429 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:13:53.988396 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:14:00.419222 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:14:00.419180 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:14:04.098596 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:14:04.098555 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:14:13.107725 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:14:13.107686 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:14:22.751273 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:14:22.751241 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:14:31.281411 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:14:31.281368 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:14:33.794341 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:14:33.794304 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:14:46.195550 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:14:46.195512 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:14:53.042561 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:14:53.042523 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 14:14:53.044520 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:14:53.044499 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 14:14:55.106871 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:14:55.106837 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:15:00.152249 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:00.152218 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728215-jfbnn"] Jul 10 14:15:00.152710 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:00.152557 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72a775fe-bcff-4b86-a56e-664631c2a21b" containerName="authorino" Jul 10 14:15:00.152710 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:00.152567 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="72a775fe-bcff-4b86-a56e-664631c2a21b" containerName="authorino" Jul 10 14:15:00.152710 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:00.152628 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="72a775fe-bcff-4b86-a56e-664631c2a21b" containerName="authorino" Jul 10 14:15:00.155584 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:00.155567 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728215-jfbnn" Jul 10 14:15:00.158419 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:00.158398 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8qd8l\"" Jul 10 14:15:00.164956 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:00.164937 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728215-jfbnn" Jul 10 14:15:00.179850 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:00.179791 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728215-jfbnn"] Jul 10 14:15:00.300537 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:00.300513 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728215-jfbnn"] Jul 10 14:15:00.303359 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:15:00.303327 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52beecd1_c181_46b5_be0f_07b31ea49dd8.slice/crio-5dbad6b45b1f7ebe3fb7f7a1172a6147d921b5c175e254d11fb66fd96f9c26da WatchSource:0}: Error finding container 5dbad6b45b1f7ebe3fb7f7a1172a6147d921b5c175e254d11fb66fd96f9c26da: Status 404 returned error can't find the container with id 5dbad6b45b1f7ebe3fb7f7a1172a6147d921b5c175e254d11fb66fd96f9c26da Jul 10 14:15:00.305473 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:00.305452 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 14:15:00.629665 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:00.629576 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728215-jfbnn" event={"ID":"52beecd1-c181-46b5-be0f-07b31ea49dd8","Type":"ContainerStarted","Data":"5dbad6b45b1f7ebe3fb7f7a1172a6147d921b5c175e254d11fb66fd96f9c26da"} Jul 10 14:15:03.650216 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:03.650174 2576 generic.go:356] "Generic (PLEG): container finished" podID="52beecd1-c181-46b5-be0f-07b31ea49dd8" containerID="7de79b8715773fb153653e004aca7f6abcae9bcb608e62fee106eedd68084920" exitCode=0 Jul 10 14:15:03.650642 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:03.650236 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728215-jfbnn" event={"ID":"52beecd1-c181-46b5-be0f-07b31ea49dd8","Type":"ContainerDied","Data":"7de79b8715773fb153653e004aca7f6abcae9bcb608e62fee106eedd68084920"} Jul 10 14:15:04.789099 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:04.789066 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728215-jfbnn" Jul 10 14:15:05.660020 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:05.659985 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728215-jfbnn" Jul 10 14:15:05.660020 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:05.659994 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728215-jfbnn" event={"ID":"52beecd1-c181-46b5-be0f-07b31ea49dd8","Type":"ContainerDied","Data":"5dbad6b45b1f7ebe3fb7f7a1172a6147d921b5c175e254d11fb66fd96f9c26da"} Jul 10 14:15:05.660402 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:05.660040 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dbad6b45b1f7ebe3fb7f7a1172a6147d921b5c175e254d11fb66fd96f9c26da" Jul 10 14:15:07.207434 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:07.207396 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:15:17.806914 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:17.806828 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:15:29.500321 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:29.500282 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:15:32.995451 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:32.995417 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:15:37.358559 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:37.358519 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:15:42.303578 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:42.303541 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:15:45.609618 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:45.609585 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:15:50.805460 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:50.805418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:15:53.756209 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:53.756175 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:15:53.851394 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:53.851349 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:15:54.109574 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:54.109493 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:15:54.227302 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:15:54.227270 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:16:00.908632 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:00.908594 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:16:01.039505 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.039467 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:16:01.086001 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.085960 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229"] Jul 10 14:16:01.086370 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.086352 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52beecd1-c181-46b5-be0f-07b31ea49dd8" containerName="cleanup" Jul 10 14:16:01.086462 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.086372 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="52beecd1-c181-46b5-be0f-07b31ea49dd8" containerName="cleanup" Jul 10 14:16:01.086462 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.086455 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="52beecd1-c181-46b5-be0f-07b31ea49dd8" containerName="cleanup" Jul 10 14:16:01.089531 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.089504 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.094199 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.094175 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b78b9ecf-openshift-default-dockercfg-w9ndk\"" Jul 10 14:16:01.108439 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.108399 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229"] Jul 10 14:16:01.139378 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.139346 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-credential-socket\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.139573 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.139388 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-workload-socket\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.139573 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.139406 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-envoy\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.139573 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.139487 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-workload-certs\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.139573 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.139520 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-token\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.139573 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.139543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-podinfo\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.139573 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.139569 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f645r\" (UniqueName: \"kubernetes.io/projected/44caa104-9ee3-4e2d-828b-714be23ba2f0-kube-api-access-f645r\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.139832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.139619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44caa104-9ee3-4e2d-828b-714be23ba2f0-istiod-ca-cert\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.139832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.139653 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-data\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.193015 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.192930 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:16:01.240232 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.240193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-workload-socket\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.240232 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.240229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-envoy\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.240448 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.240372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-workload-certs\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.241003 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.240978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-token\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.241112 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.241040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-podinfo\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.241112 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.241093 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f645r\" (UniqueName: \"kubernetes.io/projected/44caa104-9ee3-4e2d-828b-714be23ba2f0-kube-api-access-f645r\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.241223 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.241157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44caa104-9ee3-4e2d-828b-714be23ba2f0-istiod-ca-cert\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.241223 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.241164 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-workload-socket\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.241329 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.241230 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-data\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.241329 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.241294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-credential-socket\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.241431 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.241310 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-workload-certs\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.246356 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.241950 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44caa104-9ee3-4e2d-828b-714be23ba2f0-istiod-ca-cert\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.246356 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.242271 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-credential-socket\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.246356 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.242489 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-data\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.246356 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.243382 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-envoy\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.250900 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.250799 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-podinfo\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.262959 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.262933 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-token\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.265917 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.265884 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f645r\" (UniqueName: \"kubernetes.io/projected/44caa104-9ee3-4e2d-828b-714be23ba2f0-kube-api-access-f645r\") pod \"e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.403492 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.403447 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:01.547105 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.547080 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229"] Jul 10 14:16:01.548560 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:16:01.548537 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44caa104_9ee3_4e2d_828b_714be23ba2f0.slice/crio-d0b1b9da4d5f56c50d88af8b0f3dd8c3b30eda9f66e07c3dfe8b2610d14bf164 WatchSource:0}: Error finding container d0b1b9da4d5f56c50d88af8b0f3dd8c3b30eda9f66e07c3dfe8b2610d14bf164: Status 404 returned error can't find the container with id d0b1b9da4d5f56c50d88af8b0f3dd8c3b30eda9f66e07c3dfe8b2610d14bf164 Jul 10 14:16:01.550691 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.550659 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:16:01.550774 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.550729 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:16:01.550774 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.550759 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:16:01.921044 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.920993 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" event={"ID":"44caa104-9ee3-4e2d-828b-714be23ba2f0","Type":"ContainerStarted","Data":"5894b142871a3ab7f5db3bf07e4932a6db51d008749c923be8c7de724df103be"} Jul 10 14:16:01.921703 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.921067 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" event={"ID":"44caa104-9ee3-4e2d-828b-714be23ba2f0","Type":"ContainerStarted","Data":"d0b1b9da4d5f56c50d88af8b0f3dd8c3b30eda9f66e07c3dfe8b2610d14bf164"} Jul 10 14:16:01.944489 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:01.944433 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" podStartSLOduration=0.944414836 podStartE2EDuration="944.414836ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:16:01.942489066 +0000 UTC m=+1569.406094536" watchObservedRunningTime="2026-07-10 14:16:01.944414836 +0000 UTC m=+1569.408020285" Jul 10 14:16:02.243969 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:02.243879 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:16:02.361212 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:02.361174 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:16:02.404719 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:02.404675 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:02.409852 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:02.409795 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:02.478703 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:02.478668 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:16:02.925957 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:02.925926 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:02.926926 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:02.926905 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:02.947861 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:02.947827 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:16:03.088406 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:03.088369 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:16:04.914395 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:04.914353 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:16:05.870211 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:05.870171 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:16:07.289381 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:07.289343 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8"] Jul 10 14:16:07.292960 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:07.292939 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8" Jul 10 14:16:07.295873 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:07.295847 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 14:16:07.296012 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:07.295848 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8qd8l\"" Jul 10 14:16:07.305179 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:07.305146 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8"] Jul 10 14:16:07.401986 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:07.401946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22ef744-d8c2-404e-b01f-b93e4e894a16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8\" (UID: \"d22ef744-d8c2-404e-b01f-b93e4e894a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8" Jul 10 14:16:07.502506 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:07.502464 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22ef744-d8c2-404e-b01f-b93e4e894a16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8\" (UID: \"d22ef744-d8c2-404e-b01f-b93e4e894a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8" Jul 10 14:16:07.503128 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:07.503109 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22ef744-d8c2-404e-b01f-b93e4e894a16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8\" (UID: \"d22ef744-d8c2-404e-b01f-b93e4e894a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8" Jul 10 14:16:07.603967 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:07.603876 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8" Jul 10 14:16:07.790463 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:07.790438 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8"] Jul 10 14:16:07.793026 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:16:07.792997 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22ef744_d8c2_404e_b01f_b93e4e894a16.slice/crio-a3e0d082a062e23dab1dcc52794a72088890f82d2a944ce03da9f2ef79166d53 WatchSource:0}: Error finding container a3e0d082a062e23dab1dcc52794a72088890f82d2a944ce03da9f2ef79166d53: Status 404 returned error can't find the container with id a3e0d082a062e23dab1dcc52794a72088890f82d2a944ce03da9f2ef79166d53 Jul 10 14:16:07.967786 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:07.967748 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8" event={"ID":"d22ef744-d8c2-404e-b01f-b93e4e894a16","Type":"ContainerStarted","Data":"a3e0d082a062e23dab1dcc52794a72088890f82d2a944ce03da9f2ef79166d53"} Jul 10 14:16:08.973735 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:08.973692 2576 generic.go:356] "Generic (PLEG): container finished" podID="d22ef744-d8c2-404e-b01f-b93e4e894a16" containerID="fac85cf46a3d8604c16ba6bf2e3183ff573bf23e829f75b4caa43a19cf2d12c5" exitCode=7 Jul 10 14:16:08.974200 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:08.973765 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8" event={"ID":"d22ef744-d8c2-404e-b01f-b93e4e894a16","Type":"ContainerDied","Data":"fac85cf46a3d8604c16ba6bf2e3183ff573bf23e829f75b4caa43a19cf2d12c5"} Jul 10 14:16:08.974200 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:08.973985 2576 scope.go:117] "RemoveContainer" containerID="fac85cf46a3d8604c16ba6bf2e3183ff573bf23e829f75b4caa43a19cf2d12c5" Jul 10 14:16:09.979959 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:09.979919 2576 generic.go:356] "Generic (PLEG): container finished" podID="d22ef744-d8c2-404e-b01f-b93e4e894a16" containerID="0a1bdb2a2114ee960f80f2e976b721fd59d0797c0666f63b1aa1e1d7bac71633" exitCode=7 Jul 10 14:16:09.980425 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:09.979974 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8" event={"ID":"d22ef744-d8c2-404e-b01f-b93e4e894a16","Type":"ContainerDied","Data":"0a1bdb2a2114ee960f80f2e976b721fd59d0797c0666f63b1aa1e1d7bac71633"} Jul 10 14:16:09.980425 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:09.980036 2576 scope.go:117] "RemoveContainer" containerID="fac85cf46a3d8604c16ba6bf2e3183ff573bf23e829f75b4caa43a19cf2d12c5" Jul 10 14:16:09.980425 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:09.980182 2576 scope.go:117] "RemoveContainer" containerID="0a1bdb2a2114ee960f80f2e976b721fd59d0797c0666f63b1aa1e1d7bac71633" Jul 10 14:16:09.980558 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:16:09.980435 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8_odh-ai-gateway-infra(d22ef744-d8c2-404e-b01f-b93e4e894a16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8" podUID="d22ef744-d8c2-404e-b01f-b93e4e894a16" Jul 10 14:16:10.988004 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:10.987974 2576 scope.go:117] "RemoveContainer" containerID="0a1bdb2a2114ee960f80f2e976b721fd59d0797c0666f63b1aa1e1d7bac71633" Jul 10 14:16:10.988439 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:16:10.988234 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8_odh-ai-gateway-infra(d22ef744-d8c2-404e-b01f-b93e4e894a16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8" podUID="d22ef744-d8c2-404e-b01f-b93e4e894a16" Jul 10 14:16:12.998788 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:12.998755 2576 generic.go:356] "Generic (PLEG): container finished" podID="44caa104-9ee3-4e2d-828b-714be23ba2f0" containerID="5894b142871a3ab7f5db3bf07e4932a6db51d008749c923be8c7de724df103be" exitCode=0 Jul 10 14:16:12.999238 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:12.998841 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" event={"ID":"44caa104-9ee3-4e2d-828b-714be23ba2f0","Type":"ContainerDied","Data":"5894b142871a3ab7f5db3bf07e4932a6db51d008749c923be8c7de724df103be"} Jul 10 14:16:12.999279 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:12.999270 2576 scope.go:117] "RemoveContainer" containerID="5894b142871a3ab7f5db3bf07e4932a6db51d008749c923be8c7de724df103be" Jul 10 14:16:13.000395 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:13.000364 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:16:13.000466 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:13.000450 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:16:13.000513 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:13.000497 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:16:14.005342 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:14.005308 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" event={"ID":"44caa104-9ee3-4e2d-828b-714be23ba2f0","Type":"ContainerStarted","Data":"5c46558cdd9b4b6729b92dc24bb8c4ebcb2d5a26179a12a41051d9ccbd6cdd87"} Jul 10 14:16:14.403830 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:14.403768 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:14.404764 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:14.404740 2576 patch_prober.go:28] interesting pod/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 10 14:16:14.404906 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:14.404799 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" podUID="44caa104-9ee3-4e2d-828b-714be23ba2f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 10 14:16:15.404727 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:15.404687 2576 patch_prober.go:28] interesting pod/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 10 14:16:15.405166 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:15.404757 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" podUID="44caa104-9ee3-4e2d-828b-714be23ba2f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 10 14:16:16.404512 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:16.404468 2576 patch_prober.go:28] interesting pod/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 10 14:16:16.404706 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:16.404534 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" podUID="44caa104-9ee3-4e2d-828b-714be23ba2f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 10 14:16:17.404343 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:17.404299 2576 patch_prober.go:28] interesting pod/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 10 14:16:17.404878 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:17.404366 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" podUID="44caa104-9ee3-4e2d-828b-714be23ba2f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 10 14:16:17.926569 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:17.926534 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:18.408005 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:18.407971 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:18.409253 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:18.409231 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:16:23.032427 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:23.032403 2576 scope.go:117] "RemoveContainer" containerID="0a1bdb2a2114ee960f80f2e976b721fd59d0797c0666f63b1aa1e1d7bac71633" Jul 10 14:16:23.754704 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:16:23.754666 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22ef744_d8c2_404e_b01f_b93e4e894a16.slice/crio-conmon-2f54dc40fa50ac2272680972202955a06c66846a2be366941052d4d963e4f622.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22ef744_d8c2_404e_b01f_b93e4e894a16.slice/crio-2f54dc40fa50ac2272680972202955a06c66846a2be366941052d4d963e4f622.scope\": RecentStats: unable to find data in memory cache]" Jul 10 14:16:24.050396 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:24.050307 2576 generic.go:356] "Generic (PLEG): container finished" podID="d22ef744-d8c2-404e-b01f-b93e4e894a16" containerID="2f54dc40fa50ac2272680972202955a06c66846a2be366941052d4d963e4f622" exitCode=0 Jul 10 14:16:24.050396 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:24.050355 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8" event={"ID":"d22ef744-d8c2-404e-b01f-b93e4e894a16","Type":"ContainerDied","Data":"2f54dc40fa50ac2272680972202955a06c66846a2be366941052d4d963e4f622"} Jul 10 14:16:24.050896 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:24.050403 2576 scope.go:117] "RemoveContainer" containerID="0a1bdb2a2114ee960f80f2e976b721fd59d0797c0666f63b1aa1e1d7bac71633" Jul 10 14:16:25.085587 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:25.085545 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8"] Jul 10 14:16:25.192529 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:25.192502 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8" Jul 10 14:16:25.263163 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:25.263129 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22ef744-d8c2-404e-b01f-b93e4e894a16-maas-api-service-ca\") pod \"d22ef744-d8c2-404e-b01f-b93e4e894a16\" (UID: \"d22ef744-d8c2-404e-b01f-b93e4e894a16\") " Jul 10 14:16:25.263505 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:25.263480 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22ef744-d8c2-404e-b01f-b93e4e894a16-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d22ef744-d8c2-404e-b01f-b93e4e894a16" (UID: "d22ef744-d8c2-404e-b01f-b93e4e894a16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:16:25.364503 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:25.364416 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22ef744-d8c2-404e-b01f-b93e4e894a16-maas-api-service-ca\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:16:26.062369 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:26.062340 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8" Jul 10 14:16:26.062369 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:26.062343 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8" event={"ID":"d22ef744-d8c2-404e-b01f-b93e4e894a16","Type":"ContainerDied","Data":"a3e0d082a062e23dab1dcc52794a72088890f82d2a944ce03da9f2ef79166d53"} Jul 10 14:16:26.062633 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:26.062394 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3e0d082a062e23dab1dcc52794a72088890f82d2a944ce03da9f2ef79166d53" Jul 10 14:16:26.093348 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:26.093309 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8"] Jul 10 14:16:26.098760 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:26.098729 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-bqlg8"] Jul 10 14:16:27.036617 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:27.036582 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d22ef744-d8c2-404e-b01f-b93e4e894a16" path="/var/lib/kubelet/pods/d22ef744-d8c2-404e-b01f-b93e4e894a16/volumes" Jul 10 14:16:46.524798 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:46.524700 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:16:57.330748 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:57.330706 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9"] Jul 10 14:16:57.331145 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:57.331069 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d22ef744-d8c2-404e-b01f-b93e4e894a16" containerName="revoke-keys" Jul 10 14:16:57.331145 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:57.331081 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d22ef744-d8c2-404e-b01f-b93e4e894a16" containerName="revoke-keys" Jul 10 14:16:57.331145 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:57.331090 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d22ef744-d8c2-404e-b01f-b93e4e894a16" containerName="revoke-keys" Jul 10 14:16:57.331145 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:57.331095 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d22ef744-d8c2-404e-b01f-b93e4e894a16" containerName="revoke-keys" Jul 10 14:16:57.331278 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:57.331166 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d22ef744-d8c2-404e-b01f-b93e4e894a16" containerName="revoke-keys" Jul 10 14:16:57.331278 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:57.331177 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d22ef744-d8c2-404e-b01f-b93e4e894a16" containerName="revoke-keys" Jul 10 14:16:57.334091 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:57.334074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9" Jul 10 14:16:57.336676 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:57.336653 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 14:16:57.336839 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:57.336689 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8qd8l\"" Jul 10 14:16:57.342192 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:57.342159 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9"] Jul 10 14:16:57.451388 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:57.451346 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9\" (UID: \"bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9" Jul 10 14:16:57.552739 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:57.552697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9\" (UID: \"bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9" Jul 10 14:16:57.553393 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:57.553374 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9\" (UID: \"bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9" Jul 10 14:16:57.645373 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:57.645345 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9" Jul 10 14:16:57.782968 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:57.782942 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9"] Jul 10 14:16:57.785369 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:16:57.785338 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd3c27f_4e0b_4a5a_b140_f6b27d5ada36.slice/crio-840cfc4eb112b3e3aa3ea33870aac74cd3c25b158bd6cf267b04a4ffcfd59381 WatchSource:0}: Error finding container 840cfc4eb112b3e3aa3ea33870aac74cd3c25b158bd6cf267b04a4ffcfd59381: Status 404 returned error can't find the container with id 840cfc4eb112b3e3aa3ea33870aac74cd3c25b158bd6cf267b04a4ffcfd59381 Jul 10 14:16:58.210511 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:58.210417 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9" event={"ID":"bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36","Type":"ContainerStarted","Data":"ed7b9cddeb472d8fb834de471ad59a5251779edd551388c510c8ed58aaa1eb67"} Jul 10 14:16:58.210511 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:58.210483 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9" event={"ID":"bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36","Type":"ContainerStarted","Data":"840cfc4eb112b3e3aa3ea33870aac74cd3c25b158bd6cf267b04a4ffcfd59381"} Jul 10 14:16:58.228207 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:58.228150 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9" podStartSLOduration=1.2281308850000001 podStartE2EDuration="1.228130885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:16:58.226758022 +0000 UTC m=+1625.690363468" watchObservedRunningTime="2026-07-10 14:16:58.228130885 +0000 UTC m=+1625.691736330" Jul 10 14:16:59.216149 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:59.216120 2576 generic.go:356] "Generic (PLEG): container finished" podID="bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36" containerID="ed7b9cddeb472d8fb834de471ad59a5251779edd551388c510c8ed58aaa1eb67" exitCode=0 Jul 10 14:16:59.216525 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:16:59.216206 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9" event={"ID":"bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36","Type":"ContainerDied","Data":"ed7b9cddeb472d8fb834de471ad59a5251779edd551388c510c8ed58aaa1eb67"} Jul 10 14:17:00.352385 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:00.352352 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9" Jul 10 14:17:00.477800 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:00.477706 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36-maas-api-service-ca\") pod \"bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36\" (UID: \"bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36\") " Jul 10 14:17:00.478093 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:00.478066 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36" (UID: "bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:17:00.582670 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:00.579257 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36-maas-api-service-ca\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:17:01.226415 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:01.226379 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9" event={"ID":"bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36","Type":"ContainerDied","Data":"840cfc4eb112b3e3aa3ea33870aac74cd3c25b158bd6cf267b04a4ffcfd59381"} Jul 10 14:17:01.226415 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:01.226415 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9" Jul 10 14:17:01.226618 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:01.226439 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="840cfc4eb112b3e3aa3ea33870aac74cd3c25b158bd6cf267b04a4ffcfd59381" Jul 10 14:17:07.857263 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:07.857225 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:17:08.606963 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:08.606920 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:17:10.809602 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:10.809569 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:17:12.451310 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:12.451276 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:17:19.095703 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:19.095668 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:17:19.196207 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:19.196174 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:17:22.617698 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.617657 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229"] Jul 10 14:17:22.618162 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.617874 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" podUID="44caa104-9ee3-4e2d-828b-714be23ba2f0" containerName="istio-proxy" containerID="cri-o://5c46558cdd9b4b6729b92dc24bb8c4ebcb2d5a26179a12a41051d9ccbd6cdd87" gracePeriod=30 Jul 10 14:17:22.657598 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.657559 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:17:22.916736 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.916701 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps"] Jul 10 14:17:22.917109 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.917094 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d22ef744-d8c2-404e-b01f-b93e4e894a16" containerName="revoke-keys" Jul 10 14:17:22.917156 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.917112 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d22ef744-d8c2-404e-b01f-b93e4e894a16" containerName="revoke-keys" Jul 10 14:17:22.917156 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.917136 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36" containerName="revoke-keys" Jul 10 14:17:22.917156 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.917144 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36" containerName="revoke-keys" Jul 10 14:17:22.917250 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.917199 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36" containerName="revoke-keys" Jul 10 14:17:22.917250 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.917210 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d22ef744-d8c2-404e-b01f-b93e4e894a16" containerName="revoke-keys" Jul 10 14:17:22.920581 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.920561 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:22.923451 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.923427 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-561ce10f-openshift-default-dockercfg-tbbnx\"" Jul 10 14:17:22.933842 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.933803 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps"] Jul 10 14:17:22.966747 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.966717 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:17:22.972935 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.972912 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13875841-cfe6-4972-b365-6bd3b23566a4-istio-token\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:22.973040 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.972951 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-credential-socket\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:22.973040 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.972980 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mg4j\" (UniqueName: \"kubernetes.io/projected/13875841-cfe6-4972-b365-6bd3b23566a4-kube-api-access-2mg4j\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:22.973040 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.973019 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-istio-data\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:22.973278 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.973065 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-workload-socket\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:22.973278 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.973103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-workload-certs\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:22.973278 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.973154 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13875841-cfe6-4972-b365-6bd3b23566a4-istiod-ca-cert\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:22.973278 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.973173 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-istio-envoy\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:22.973278 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:22.973190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13875841-cfe6-4972-b365-6bd3b23566a4-istio-podinfo\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.074150 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.074114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-workload-certs\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.074354 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.074161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13875841-cfe6-4972-b365-6bd3b23566a4-istiod-ca-cert\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.074354 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.074186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-istio-envoy\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.074354 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.074206 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13875841-cfe6-4972-b365-6bd3b23566a4-istio-podinfo\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.074518 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.074379 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13875841-cfe6-4972-b365-6bd3b23566a4-istio-token\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.074518 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.074443 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-credential-socket\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.074658 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.074627 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mg4j\" (UniqueName: \"kubernetes.io/projected/13875841-cfe6-4972-b365-6bd3b23566a4-kube-api-access-2mg4j\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.074726 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.074682 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-istio-data\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.074782 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.074735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-workload-socket\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.074782 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.074759 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-credential-socket\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.074910 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.074780 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-workload-certs\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.074910 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.074884 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13875841-cfe6-4972-b365-6bd3b23566a4-istiod-ca-cert\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.075047 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.075024 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-istio-data\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.075101 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.075045 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-workload-socket\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.076549 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.076530 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-istio-envoy\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.076792 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.076761 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13875841-cfe6-4972-b365-6bd3b23566a4-istio-podinfo\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.083154 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.083133 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13875841-cfe6-4972-b365-6bd3b23566a4-istio-token\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.083601 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.083581 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mg4j\" (UniqueName: \"kubernetes.io/projected/13875841-cfe6-4972-b365-6bd3b23566a4-kube-api-access-2mg4j\") pod \"e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.231784 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.231689 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:23.369262 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.369235 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps"] Jul 10 14:17:23.371143 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:17:23.371115 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13875841_cfe6_4972_b365_6bd3b23566a4.slice/crio-cf0c7f65ff3db31f3baa681c8ea4174f746365d50ca115b21f93ef01d86b53e7 WatchSource:0}: Error finding container cf0c7f65ff3db31f3baa681c8ea4174f746365d50ca115b21f93ef01d86b53e7: Status 404 returned error can't find the container with id cf0c7f65ff3db31f3baa681c8ea4174f746365d50ca115b21f93ef01d86b53e7 Jul 10 14:17:23.373166 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.373136 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:17:23.373235 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.373203 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:17:23.373235 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:23.373230 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:17:24.154987 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:24.154949 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:17:24.254753 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:24.254717 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:17:24.339888 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:24.339843 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" event={"ID":"13875841-cfe6-4972-b365-6bd3b23566a4","Type":"ContainerStarted","Data":"f6a2553c4860a889cef7df371f4d7423a32d1495ef4ad7b59f5f08881df1cbeb"} Jul 10 14:17:24.340072 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:24.339939 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" event={"ID":"13875841-cfe6-4972-b365-6bd3b23566a4","Type":"ContainerStarted","Data":"cf0c7f65ff3db31f3baa681c8ea4174f746365d50ca115b21f93ef01d86b53e7"} Jul 10 14:17:24.365007 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:24.364956 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" podStartSLOduration=2.36493571 podStartE2EDuration="2.36493571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:17:24.362600029 +0000 UTC m=+1651.826205474" watchObservedRunningTime="2026-07-10 14:17:24.36493571 +0000 UTC m=+1651.828541153" Jul 10 14:17:24.460464 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:24.460370 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:17:24.860488 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:24.860398 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:17:24.999944 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:24.999907 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:17:25.107271 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:25.107238 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:17:25.232069 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:25.232026 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:25.237347 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:25.237310 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:25.344843 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:25.344768 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:25.345930 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:25.345910 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:26.895461 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:26.895426 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:17:27.887659 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.887634 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:17:27.921233 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.921085 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f645r\" (UniqueName: \"kubernetes.io/projected/44caa104-9ee3-4e2d-828b-714be23ba2f0-kube-api-access-f645r\") pod \"44caa104-9ee3-4e2d-828b-714be23ba2f0\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " Jul 10 14:17:27.921233 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.921159 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-envoy\") pod \"44caa104-9ee3-4e2d-828b-714be23ba2f0\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " Jul 10 14:17:27.921786 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.921606 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-workload-socket\") pod \"44caa104-9ee3-4e2d-828b-714be23ba2f0\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " Jul 10 14:17:27.921786 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.921638 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-credential-socket\") pod \"44caa104-9ee3-4e2d-828b-714be23ba2f0\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " Jul 10 14:17:27.921786 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.921666 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-data\") pod \"44caa104-9ee3-4e2d-828b-714be23ba2f0\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " Jul 10 14:17:27.921786 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.921688 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44caa104-9ee3-4e2d-828b-714be23ba2f0-istiod-ca-cert\") pod \"44caa104-9ee3-4e2d-828b-714be23ba2f0\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " Jul 10 14:17:27.921786 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.921750 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-workload-certs\") pod \"44caa104-9ee3-4e2d-828b-714be23ba2f0\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " Jul 10 14:17:27.921786 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.921765 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-token\") pod \"44caa104-9ee3-4e2d-828b-714be23ba2f0\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " Jul 10 14:17:27.921786 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.921788 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-podinfo\") pod \"44caa104-9ee3-4e2d-828b-714be23ba2f0\" (UID: \"44caa104-9ee3-4e2d-828b-714be23ba2f0\") " Jul 10 14:17:27.922661 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.922450 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "44caa104-9ee3-4e2d-828b-714be23ba2f0" (UID: "44caa104-9ee3-4e2d-828b-714be23ba2f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:17:27.923880 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.922979 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "44caa104-9ee3-4e2d-828b-714be23ba2f0" (UID: "44caa104-9ee3-4e2d-828b-714be23ba2f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:17:27.923880 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.923054 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "44caa104-9ee3-4e2d-828b-714be23ba2f0" (UID: "44caa104-9ee3-4e2d-828b-714be23ba2f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:17:27.923880 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.923849 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-data" (OuterVolumeSpecName: "istio-data") pod "44caa104-9ee3-4e2d-828b-714be23ba2f0" (UID: "44caa104-9ee3-4e2d-828b-714be23ba2f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:17:27.924996 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.924959 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44caa104-9ee3-4e2d-828b-714be23ba2f0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "44caa104-9ee3-4e2d-828b-714be23ba2f0" (UID: "44caa104-9ee3-4e2d-828b-714be23ba2f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:17:27.936043 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.935025 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44caa104-9ee3-4e2d-828b-714be23ba2f0-kube-api-access-f645r" (OuterVolumeSpecName: "kube-api-access-f645r") pod "44caa104-9ee3-4e2d-828b-714be23ba2f0" (UID: "44caa104-9ee3-4e2d-828b-714be23ba2f0"). InnerVolumeSpecName "kube-api-access-f645r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:17:27.946181 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.943646 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "44caa104-9ee3-4e2d-828b-714be23ba2f0" (UID: "44caa104-9ee3-4e2d-828b-714be23ba2f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 14:17:27.946181 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.943784 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-token" (OuterVolumeSpecName: "istio-token") pod "44caa104-9ee3-4e2d-828b-714be23ba2f0" (UID: "44caa104-9ee3-4e2d-828b-714be23ba2f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:17:27.952003 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:27.951949 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "44caa104-9ee3-4e2d-828b-714be23ba2f0" (UID: "44caa104-9ee3-4e2d-828b-714be23ba2f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:17:28.042832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.035075 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-data\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:17:28.042832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.035110 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44caa104-9ee3-4e2d-828b-714be23ba2f0-istiod-ca-cert\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:17:28.042832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.035123 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-workload-certs\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:17:28.042832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.035136 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-token\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:17:28.042832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.035149 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-podinfo\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:17:28.042832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.035162 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f645r\" (UniqueName: \"kubernetes.io/projected/44caa104-9ee3-4e2d-828b-714be23ba2f0-kube-api-access-f645r\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:17:28.042832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.035175 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-istio-envoy\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:17:28.042832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.035189 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-workload-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:17:28.042832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.035201 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44caa104-9ee3-4e2d-828b-714be23ba2f0-credential-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:17:28.361797 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.361698 2576 generic.go:356] "Generic (PLEG): container finished" podID="44caa104-9ee3-4e2d-828b-714be23ba2f0" containerID="5c46558cdd9b4b6729b92dc24bb8c4ebcb2d5a26179a12a41051d9ccbd6cdd87" exitCode=0 Jul 10 14:17:28.361797 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.361757 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" Jul 10 14:17:28.361797 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.361780 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" event={"ID":"44caa104-9ee3-4e2d-828b-714be23ba2f0","Type":"ContainerDied","Data":"5c46558cdd9b4b6729b92dc24bb8c4ebcb2d5a26179a12a41051d9ccbd6cdd87"} Jul 10 14:17:28.362097 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.361859 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229" event={"ID":"44caa104-9ee3-4e2d-828b-714be23ba2f0","Type":"ContainerDied","Data":"d0b1b9da4d5f56c50d88af8b0f3dd8c3b30eda9f66e07c3dfe8b2610d14bf164"} Jul 10 14:17:28.362097 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.361880 2576 scope.go:117] "RemoveContainer" containerID="5c46558cdd9b4b6729b92dc24bb8c4ebcb2d5a26179a12a41051d9ccbd6cdd87" Jul 10 14:17:28.371091 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.371070 2576 scope.go:117] "RemoveContainer" containerID="5894b142871a3ab7f5db3bf07e4932a6db51d008749c923be8c7de724df103be" Jul 10 14:17:28.379597 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.379574 2576 scope.go:117] "RemoveContainer" containerID="5c46558cdd9b4b6729b92dc24bb8c4ebcb2d5a26179a12a41051d9ccbd6cdd87" Jul 10 14:17:28.379931 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:17:28.379912 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c46558cdd9b4b6729b92dc24bb8c4ebcb2d5a26179a12a41051d9ccbd6cdd87\": container with ID starting with 5c46558cdd9b4b6729b92dc24bb8c4ebcb2d5a26179a12a41051d9ccbd6cdd87 not found: ID does not exist" containerID="5c46558cdd9b4b6729b92dc24bb8c4ebcb2d5a26179a12a41051d9ccbd6cdd87" Jul 10 14:17:28.380002 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.379943 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c46558cdd9b4b6729b92dc24bb8c4ebcb2d5a26179a12a41051d9ccbd6cdd87"} err="failed to get container status \"5c46558cdd9b4b6729b92dc24bb8c4ebcb2d5a26179a12a41051d9ccbd6cdd87\": rpc error: code = NotFound desc = could not find container \"5c46558cdd9b4b6729b92dc24bb8c4ebcb2d5a26179a12a41051d9ccbd6cdd87\": container with ID starting with 5c46558cdd9b4b6729b92dc24bb8c4ebcb2d5a26179a12a41051d9ccbd6cdd87 not found: ID does not exist" Jul 10 14:17:28.380002 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.379961 2576 scope.go:117] "RemoveContainer" containerID="5894b142871a3ab7f5db3bf07e4932a6db51d008749c923be8c7de724df103be" Jul 10 14:17:28.380210 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:17:28.380192 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5894b142871a3ab7f5db3bf07e4932a6db51d008749c923be8c7de724df103be\": container with ID starting with 5894b142871a3ab7f5db3bf07e4932a6db51d008749c923be8c7de724df103be not found: ID does not exist" containerID="5894b142871a3ab7f5db3bf07e4932a6db51d008749c923be8c7de724df103be" Jul 10 14:17:28.380255 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.380212 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5894b142871a3ab7f5db3bf07e4932a6db51d008749c923be8c7de724df103be"} err="failed to get container status \"5894b142871a3ab7f5db3bf07e4932a6db51d008749c923be8c7de724df103be\": rpc error: code = NotFound desc = could not find container \"5894b142871a3ab7f5db3bf07e4932a6db51d008749c923be8c7de724df103be\": container with ID starting with 5894b142871a3ab7f5db3bf07e4932a6db51d008749c923be8c7de724df103be not found: ID does not exist" Jul 10 14:17:28.384760 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.384731 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229"] Jul 10 14:17:28.389498 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:28.389472 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b78b9ecf-openshift-default-75c4df4bdc-rp229"] Jul 10 14:17:29.044345 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:29.043946 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44caa104-9ee3-4e2d-828b-714be23ba2f0" path="/var/lib/kubelet/pods/44caa104-9ee3-4e2d-828b-714be23ba2f0/volumes" Jul 10 14:17:29.152701 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:29.152668 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv"] Jul 10 14:17:29.153052 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:29.153039 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44caa104-9ee3-4e2d-828b-714be23ba2f0" containerName="istio-proxy" Jul 10 14:17:29.153096 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:29.153054 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44caa104-9ee3-4e2d-828b-714be23ba2f0" containerName="istio-proxy" Jul 10 14:17:29.153136 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:29.153127 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="44caa104-9ee3-4e2d-828b-714be23ba2f0" containerName="istio-proxy" Jul 10 14:17:29.153172 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:29.153138 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="44caa104-9ee3-4e2d-828b-714be23ba2f0" containerName="istio-proxy" Jul 10 14:17:29.155429 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:29.155410 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv" Jul 10 14:17:29.158040 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:29.158014 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 14:17:29.158198 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:29.158179 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8qd8l\"" Jul 10 14:17:29.168684 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:29.168649 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv"] Jul 10 14:17:29.248466 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:29.248411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d6f163a-87c3-4ed8-8420-af54862a2e22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv\" (UID: \"8d6f163a-87c3-4ed8-8420-af54862a2e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv" Jul 10 14:17:29.349383 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:29.349288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d6f163a-87c3-4ed8-8420-af54862a2e22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv\" (UID: \"8d6f163a-87c3-4ed8-8420-af54862a2e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv" Jul 10 14:17:29.349972 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:29.349949 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d6f163a-87c3-4ed8-8420-af54862a2e22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv\" (UID: \"8d6f163a-87c3-4ed8-8420-af54862a2e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv" Jul 10 14:17:29.473602 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:29.473561 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv" Jul 10 14:17:29.648831 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:29.648417 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv"] Jul 10 14:17:29.649181 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:17:29.649151 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6f163a_87c3_4ed8_8420_af54862a2e22.slice/crio-0f1ec18813c211261ab25cf8335fa0a8e93d2d225c22791b65f116012430c247 WatchSource:0}: Error finding container 0f1ec18813c211261ab25cf8335fa0a8e93d2d225c22791b65f116012430c247: Status 404 returned error can't find the container with id 0f1ec18813c211261ab25cf8335fa0a8e93d2d225c22791b65f116012430c247 Jul 10 14:17:30.375399 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:30.375309 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv" event={"ID":"8d6f163a-87c3-4ed8-8420-af54862a2e22","Type":"ContainerStarted","Data":"bf53c4bc1ea232880b5e9a968e236ec4609ee27fb3b29930b77ba708f6b57b25"} Jul 10 14:17:30.375399 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:30.375361 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv" event={"ID":"8d6f163a-87c3-4ed8-8420-af54862a2e22","Type":"ContainerStarted","Data":"0f1ec18813c211261ab25cf8335fa0a8e93d2d225c22791b65f116012430c247"} Jul 10 14:17:30.395330 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:30.395276 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv" podStartSLOduration=1.3952582 podStartE2EDuration="1.3952582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:17:30.394143575 +0000 UTC m=+1657.857749022" watchObservedRunningTime="2026-07-10 14:17:30.3952582 +0000 UTC m=+1657.858863623" Jul 10 14:17:31.381444 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:31.381410 2576 generic.go:356] "Generic (PLEG): container finished" podID="8d6f163a-87c3-4ed8-8420-af54862a2e22" containerID="bf53c4bc1ea232880b5e9a968e236ec4609ee27fb3b29930b77ba708f6b57b25" exitCode=7 Jul 10 14:17:31.381852 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:31.381454 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv" event={"ID":"8d6f163a-87c3-4ed8-8420-af54862a2e22","Type":"ContainerDied","Data":"bf53c4bc1ea232880b5e9a968e236ec4609ee27fb3b29930b77ba708f6b57b25"} Jul 10 14:17:31.381852 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:31.381708 2576 scope.go:117] "RemoveContainer" containerID="bf53c4bc1ea232880b5e9a968e236ec4609ee27fb3b29930b77ba708f6b57b25" Jul 10 14:17:32.388684 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:32.388590 2576 generic.go:356] "Generic (PLEG): container finished" podID="8d6f163a-87c3-4ed8-8420-af54862a2e22" containerID="f6d1928d90f9777e08d973b61fd0083e59705e07833aae2007c30360e246ef01" exitCode=7 Jul 10 14:17:32.389128 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:32.388680 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv" event={"ID":"8d6f163a-87c3-4ed8-8420-af54862a2e22","Type":"ContainerDied","Data":"f6d1928d90f9777e08d973b61fd0083e59705e07833aae2007c30360e246ef01"} Jul 10 14:17:32.389128 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:32.388752 2576 scope.go:117] "RemoveContainer" containerID="bf53c4bc1ea232880b5e9a968e236ec4609ee27fb3b29930b77ba708f6b57b25" Jul 10 14:17:32.389128 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:32.388950 2576 scope.go:117] "RemoveContainer" containerID="f6d1928d90f9777e08d973b61fd0083e59705e07833aae2007c30360e246ef01" Jul 10 14:17:32.389269 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:17:32.389186 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv_odh-ai-gateway-infra(8d6f163a-87c3-4ed8-8420-af54862a2e22)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv" podUID="8d6f163a-87c3-4ed8-8420-af54862a2e22" Jul 10 14:17:33.395061 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:33.395032 2576 scope.go:117] "RemoveContainer" containerID="f6d1928d90f9777e08d973b61fd0083e59705e07833aae2007c30360e246ef01" Jul 10 14:17:33.395530 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:17:33.395261 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv_odh-ai-gateway-infra(8d6f163a-87c3-4ed8-8420-af54862a2e22)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv" podUID="8d6f163a-87c3-4ed8-8420-af54862a2e22" Jul 10 14:17:34.524430 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:34.524391 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:17:34.641273 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:34.641224 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:17:35.406237 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:35.406201 2576 generic.go:356] "Generic (PLEG): container finished" podID="13875841-cfe6-4972-b365-6bd3b23566a4" containerID="f6a2553c4860a889cef7df371f4d7423a32d1495ef4ad7b59f5f08881df1cbeb" exitCode=0 Jul 10 14:17:35.406419 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:35.406259 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" event={"ID":"13875841-cfe6-4972-b365-6bd3b23566a4","Type":"ContainerDied","Data":"f6a2553c4860a889cef7df371f4d7423a32d1495ef4ad7b59f5f08881df1cbeb"} Jul 10 14:17:35.406715 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:35.406697 2576 scope.go:117] "RemoveContainer" containerID="f6a2553c4860a889cef7df371f4d7423a32d1495ef4ad7b59f5f08881df1cbeb" Jul 10 14:17:35.407673 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:35.407645 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:17:35.407771 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:35.407712 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:17:35.407771 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:35.407750 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:17:36.412936 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:36.412900 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" event={"ID":"13875841-cfe6-4972-b365-6bd3b23566a4","Type":"ContainerStarted","Data":"e49d8e5d4b7a356f4698b93df7ab260f54859369afb905c3adddc17ac5160062"} Jul 10 14:17:37.232388 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:37.232345 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:37.233702 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:37.233665 2576 patch_prober.go:28] interesting pod/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 10 14:17:37.233889 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:37.233734 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" podUID="13875841-cfe6-4972-b365-6bd3b23566a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 10 14:17:38.117866 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:38.117832 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:17:38.233190 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:38.233146 2576 patch_prober.go:28] interesting pod/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 10 14:17:38.233520 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:38.233483 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" podUID="13875841-cfe6-4972-b365-6bd3b23566a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 10 14:17:39.233188 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:39.233140 2576 patch_prober.go:28] interesting pod/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 10 14:17:39.233718 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:39.233213 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" podUID="13875841-cfe6-4972-b365-6bd3b23566a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 10 14:17:40.233305 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:40.233259 2576 patch_prober.go:28] interesting pod/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 10 14:17:40.233975 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:40.233939 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" podUID="13875841-cfe6-4972-b365-6bd3b23566a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 10 14:17:40.345000 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:40.344960 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:41.236881 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:41.236849 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:41.237827 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:41.237794 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:17:45.031025 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:45.030992 2576 scope.go:117] "RemoveContainer" containerID="f6d1928d90f9777e08d973b61fd0083e59705e07833aae2007c30360e246ef01" Jul 10 14:17:45.476473 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:45.476440 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv" event={"ID":"8d6f163a-87c3-4ed8-8420-af54862a2e22","Type":"ContainerStarted","Data":"c701b4121933c38ca9a7ff30be4777ccd91e16b952c9c867f3e54cccc6c8ab32"} Jul 10 14:17:46.482901 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:46.482862 2576 generic.go:356] "Generic (PLEG): container finished" podID="8d6f163a-87c3-4ed8-8420-af54862a2e22" containerID="c701b4121933c38ca9a7ff30be4777ccd91e16b952c9c867f3e54cccc6c8ab32" exitCode=0 Jul 10 14:17:46.483342 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:46.482924 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv" event={"ID":"8d6f163a-87c3-4ed8-8420-af54862a2e22","Type":"ContainerDied","Data":"c701b4121933c38ca9a7ff30be4777ccd91e16b952c9c867f3e54cccc6c8ab32"} Jul 10 14:17:46.483342 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:46.483004 2576 scope.go:117] "RemoveContainer" containerID="f6d1928d90f9777e08d973b61fd0083e59705e07833aae2007c30360e246ef01" Jul 10 14:17:46.506142 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:46.506103 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv"] Jul 10 14:17:47.627902 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:47.627877 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv" Jul 10 14:17:47.723973 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:47.723938 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d6f163a-87c3-4ed8-8420-af54862a2e22-maas-api-service-ca\") pod \"8d6f163a-87c3-4ed8-8420-af54862a2e22\" (UID: \"8d6f163a-87c3-4ed8-8420-af54862a2e22\") " Jul 10 14:17:47.724348 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:47.724323 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d6f163a-87c3-4ed8-8420-af54862a2e22-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8d6f163a-87c3-4ed8-8420-af54862a2e22" (UID: "8d6f163a-87c3-4ed8-8420-af54862a2e22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:17:47.824568 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:47.824483 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d6f163a-87c3-4ed8-8420-af54862a2e22-maas-api-service-ca\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:17:48.494676 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:48.494646 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv" Jul 10 14:17:48.494676 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:48.494657 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv" event={"ID":"8d6f163a-87c3-4ed8-8420-af54862a2e22","Type":"ContainerDied","Data":"0f1ec18813c211261ab25cf8335fa0a8e93d2d225c22791b65f116012430c247"} Jul 10 14:17:48.494939 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:48.494702 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f1ec18813c211261ab25cf8335fa0a8e93d2d225c22791b65f116012430c247" Jul 10 14:17:48.525606 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:48.525557 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv"] Jul 10 14:17:48.531571 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:48.531537 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-4p5nv"] Jul 10 14:17:49.036633 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:49.036601 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d6f163a-87c3-4ed8-8420-af54862a2e22" path="/var/lib/kubelet/pods/8d6f163a-87c3-4ed8-8420-af54862a2e22/volumes" Jul 10 14:17:49.617337 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:49.617303 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:17:49.727056 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:17:49.727022 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:18:19.251242 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:19.251149 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-bgm49"] Jul 10 14:18:19.251731 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:19.251505 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44caa104-9ee3-4e2d-828b-714be23ba2f0" containerName="istio-proxy" Jul 10 14:18:19.251731 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:19.251516 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44caa104-9ee3-4e2d-828b-714be23ba2f0" containerName="istio-proxy" Jul 10 14:18:19.251731 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:19.251524 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d6f163a-87c3-4ed8-8420-af54862a2e22" containerName="revoke-keys" Jul 10 14:18:19.251731 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:19.251531 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d6f163a-87c3-4ed8-8420-af54862a2e22" containerName="revoke-keys" Jul 10 14:18:19.251731 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:19.251540 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d6f163a-87c3-4ed8-8420-af54862a2e22" containerName="revoke-keys" Jul 10 14:18:19.251731 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:19.251547 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d6f163a-87c3-4ed8-8420-af54862a2e22" containerName="revoke-keys" Jul 10 14:18:19.251731 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:19.251614 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d6f163a-87c3-4ed8-8420-af54862a2e22" containerName="revoke-keys" Jul 10 14:18:19.251731 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:19.251628 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d6f163a-87c3-4ed8-8420-af54862a2e22" containerName="revoke-keys" Jul 10 14:18:19.254593 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:19.254574 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-bgm49" Jul 10 14:18:19.259103 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:18:19.259073 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-ait-561ce10f-bgm49\" is forbidden: User \"system:node:ip-10-0-143-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-143-25.ec2.internal' and this object" podUID="49293a72-c240-4d30-9978-82945f1dedc6" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-bgm49" Jul 10 14:18:19.259263 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:19.259241 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 14:18:19.259316 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:19.259284 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8qd8l\"" Jul 10 14:18:19.279556 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:19.279522 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-bgm49"] Jul 10 14:18:19.300828 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:19.300771 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49293a72-c240-4d30-9978-82945f1dedc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-561ce10f-bgm49\" (UID: \"49293a72-c240-4d30-9978-82945f1dedc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-bgm49" Jul 10 14:18:19.402280 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:19.402237 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49293a72-c240-4d30-9978-82945f1dedc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-561ce10f-bgm49\" (UID: \"49293a72-c240-4d30-9978-82945f1dedc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-bgm49" Jul 10 14:18:19.402910 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:19.402891 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49293a72-c240-4d30-9978-82945f1dedc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-561ce10f-bgm49\" (UID: \"49293a72-c240-4d30-9978-82945f1dedc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-bgm49" Jul 10 14:18:19.564787 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:19.564692 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-bgm49" Jul 10 14:18:19.700363 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:19.700333 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-bgm49"] Jul 10 14:18:19.703678 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:18:19.703654 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49293a72_c240_4d30_9978_82945f1dedc6.slice/crio-c57d757978b68a558c68b836090531647e5fda2f2bbcafe2dcf6c4698047225a WatchSource:0}: Error finding container c57d757978b68a558c68b836090531647e5fda2f2bbcafe2dcf6c4698047225a: Status 404 returned error can't find the container with id c57d757978b68a558c68b836090531647e5fda2f2bbcafe2dcf6c4698047225a Jul 10 14:18:20.645408 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:20.645367 2576 generic.go:356] "Generic (PLEG): container finished" podID="49293a72-c240-4d30-9978-82945f1dedc6" containerID="909d97d1e683a70b727e09882c5a16d1326ce0eb54363ca3dcc09d5e30eb5fab" exitCode=0 Jul 10 14:18:20.645870 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:20.645452 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-bgm49" event={"ID":"49293a72-c240-4d30-9978-82945f1dedc6","Type":"ContainerDied","Data":"909d97d1e683a70b727e09882c5a16d1326ce0eb54363ca3dcc09d5e30eb5fab"} Jul 10 14:18:20.645870 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:20.645498 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-bgm49" event={"ID":"49293a72-c240-4d30-9978-82945f1dedc6","Type":"ContainerStarted","Data":"c57d757978b68a558c68b836090531647e5fda2f2bbcafe2dcf6c4698047225a"} Jul 10 14:18:21.781474 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:21.781447 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-bgm49" Jul 10 14:18:21.825300 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:21.825270 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49293a72-c240-4d30-9978-82945f1dedc6-maas-api-service-ca\") pod \"49293a72-c240-4d30-9978-82945f1dedc6\" (UID: \"49293a72-c240-4d30-9978-82945f1dedc6\") " Jul 10 14:18:21.825668 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:21.825641 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49293a72-c240-4d30-9978-82945f1dedc6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "49293a72-c240-4d30-9978-82945f1dedc6" (UID: "49293a72-c240-4d30-9978-82945f1dedc6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:18:21.926200 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:21.926112 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49293a72-c240-4d30-9978-82945f1dedc6-maas-api-service-ca\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:18:22.656407 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:22.656376 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-bgm49" Jul 10 14:18:22.656599 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:22.656424 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-561ce10f-bgm49" event={"ID":"49293a72-c240-4d30-9978-82945f1dedc6","Type":"ContainerDied","Data":"c57d757978b68a558c68b836090531647e5fda2f2bbcafe2dcf6c4698047225a"} Jul 10 14:18:22.656599 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:22.656474 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c57d757978b68a558c68b836090531647e5fda2f2bbcafe2dcf6c4698047225a" Jul 10 14:18:29.804345 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:29.804276 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:18:32.803403 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:32.803366 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:18:34.337494 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:34.337460 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:18:40.875406 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:40.875371 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:18:40.958556 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:40.958514 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:18:45.342668 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.342632 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps"] Jul 10 14:18:45.343083 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.342848 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" podUID="13875841-cfe6-4972-b365-6bd3b23566a4" containerName="istio-proxy" containerID="cri-o://e49d8e5d4b7a356f4698b93df7ab260f54859369afb905c3adddc17ac5160062" gracePeriod=30 Jul 10 14:18:45.388671 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.388623 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:18:45.651454 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.651415 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557"] Jul 10 14:18:45.652000 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.651980 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49293a72-c240-4d30-9978-82945f1dedc6" containerName="revoke-keys" Jul 10 14:18:45.652053 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.652004 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="49293a72-c240-4d30-9978-82945f1dedc6" containerName="revoke-keys" Jul 10 14:18:45.652053 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.652037 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d6f163a-87c3-4ed8-8420-af54862a2e22" containerName="revoke-keys" Jul 10 14:18:45.652053 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.652046 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d6f163a-87c3-4ed8-8420-af54862a2e22" containerName="revoke-keys" Jul 10 14:18:45.652152 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.652124 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d6f163a-87c3-4ed8-8420-af54862a2e22" containerName="revoke-keys" Jul 10 14:18:45.652152 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.652136 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="49293a72-c240-4d30-9978-82945f1dedc6" containerName="revoke-keys" Jul 10 14:18:45.655603 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.655587 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.658579 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.658334 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-cce0783f-openshift-default-dockercfg-gbt8p\"" Jul 10 14:18:45.664502 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.664478 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557"] Jul 10 14:18:45.702218 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.702191 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:18:45.741937 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.741906 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istiod-ca-cert\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.742092 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.741944 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-workload-socket\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.742092 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.741974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfktb\" (UniqueName: \"kubernetes.io/projected/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-kube-api-access-rfktb\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.742092 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.741995 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-credential-socket\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.742249 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.742098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-podinfo\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.742249 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.742139 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-token\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.742249 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.742192 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-workload-certs\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.742249 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.742220 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-data\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.742392 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.742275 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-envoy\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.843374 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.843335 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istiod-ca-cert\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.843374 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.843376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-workload-socket\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.843632 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.843416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfktb\" (UniqueName: \"kubernetes.io/projected/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-kube-api-access-rfktb\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.843632 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.843444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-credential-socket\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.843632 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.843503 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-podinfo\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.843632 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.843526 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-token\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.843632 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.843555 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-workload-certs\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.843632 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.843588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-data\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.843632 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.843619 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-envoy\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.844060 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.844033 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-credential-socket\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.844118 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.844075 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-workload-certs\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.844118 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.844094 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-data\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.844186 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.844154 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-workload-socket\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.844235 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.844214 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istiod-ca-cert\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.846039 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.846022 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-envoy\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.846164 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.846146 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-podinfo\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.853098 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.853073 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-token\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.853352 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.853331 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfktb\" (UniqueName: \"kubernetes.io/projected/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-kube-api-access-rfktb\") pod \"e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:45.968273 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:45.968179 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:46.107763 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:46.107684 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557"] Jul 10 14:18:46.110280 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:18:46.110255 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db3ea0c_a225_42fd_ac1c_a431e6b61d59.slice/crio-f9d0d17899e703509be1ff15f18c5fed88df7ad01337fe1fe67ff85586a279be WatchSource:0}: Error finding container f9d0d17899e703509be1ff15f18c5fed88df7ad01337fe1fe67ff85586a279be: Status 404 returned error can't find the container with id f9d0d17899e703509be1ff15f18c5fed88df7ad01337fe1fe67ff85586a279be Jul 10 14:18:46.113034 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:46.112994 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:18:46.113141 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:46.113087 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:18:46.113141 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:46.113130 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:18:46.771359 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:46.771326 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" event={"ID":"2db3ea0c-a225-42fd-ac1c-a431e6b61d59","Type":"ContainerStarted","Data":"1e0420dd1331a6028fa672a5fab8b2f2fb8ff80269118deeadcff6945ae9669b"} Jul 10 14:18:46.771723 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:46.771379 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" event={"ID":"2db3ea0c-a225-42fd-ac1c-a431e6b61d59","Type":"ContainerStarted","Data":"f9d0d17899e703509be1ff15f18c5fed88df7ad01337fe1fe67ff85586a279be"} Jul 10 14:18:46.823065 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:46.823009 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" podStartSLOduration=1.822993849 podStartE2EDuration="1.822993849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:18:46.821958573 +0000 UTC m=+1734.285564066" watchObservedRunningTime="2026-07-10 14:18:46.822993849 +0000 UTC m=+1734.286599312" Jul 10 14:18:46.847909 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:46.847879 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:18:46.954362 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:46.954326 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:18:46.968254 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:46.968222 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:47.139997 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:47.139907 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:18:47.728476 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:47.728444 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:18:47.887480 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:47.887447 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:18:47.974151 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:47.974120 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:47.994280 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:47.994190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:18:48.781873 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:48.781841 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:48.782764 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:48.782744 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:49.201992 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:49.201959 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:18:50.602772 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.602707 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:18:50.689552 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.689520 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13875841-cfe6-4972-b365-6bd3b23566a4-istio-token\") pod \"13875841-cfe6-4972-b365-6bd3b23566a4\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " Jul 10 14:18:50.689732 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.689577 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-workload-certs\") pod \"13875841-cfe6-4972-b365-6bd3b23566a4\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " Jul 10 14:18:50.689732 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.689605 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-credential-socket\") pod \"13875841-cfe6-4972-b365-6bd3b23566a4\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " Jul 10 14:18:50.689732 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.689633 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-workload-socket\") pod \"13875841-cfe6-4972-b365-6bd3b23566a4\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " Jul 10 14:18:50.689732 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.689657 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13875841-cfe6-4972-b365-6bd3b23566a4-istio-podinfo\") pod \"13875841-cfe6-4972-b365-6bd3b23566a4\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " Jul 10 14:18:50.689732 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.689706 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13875841-cfe6-4972-b365-6bd3b23566a4-istiod-ca-cert\") pod \"13875841-cfe6-4972-b365-6bd3b23566a4\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " Jul 10 14:18:50.690049 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.689738 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-istio-envoy\") pod \"13875841-cfe6-4972-b365-6bd3b23566a4\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " Jul 10 14:18:50.690049 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.689781 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-istio-data\") pod \"13875841-cfe6-4972-b365-6bd3b23566a4\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " Jul 10 14:18:50.690049 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.689824 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mg4j\" (UniqueName: \"kubernetes.io/projected/13875841-cfe6-4972-b365-6bd3b23566a4-kube-api-access-2mg4j\") pod \"13875841-cfe6-4972-b365-6bd3b23566a4\" (UID: \"13875841-cfe6-4972-b365-6bd3b23566a4\") " Jul 10 14:18:50.690049 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.689939 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "13875841-cfe6-4972-b365-6bd3b23566a4" (UID: "13875841-cfe6-4972-b365-6bd3b23566a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:18:50.690049 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.689954 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "13875841-cfe6-4972-b365-6bd3b23566a4" (UID: "13875841-cfe6-4972-b365-6bd3b23566a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:18:50.690316 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.690126 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-workload-certs\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:18:50.690316 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.690148 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-workload-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:18:50.690316 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.690300 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "13875841-cfe6-4972-b365-6bd3b23566a4" (UID: "13875841-cfe6-4972-b365-6bd3b23566a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:18:50.690865 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.690826 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13875841-cfe6-4972-b365-6bd3b23566a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "13875841-cfe6-4972-b365-6bd3b23566a4" (UID: "13875841-cfe6-4972-b365-6bd3b23566a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:18:50.690988 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.690870 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "13875841-cfe6-4972-b365-6bd3b23566a4" (UID: "13875841-cfe6-4972-b365-6bd3b23566a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:18:50.692625 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.692591 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13875841-cfe6-4972-b365-6bd3b23566a4-kube-api-access-2mg4j" (OuterVolumeSpecName: "kube-api-access-2mg4j") pod "13875841-cfe6-4972-b365-6bd3b23566a4" (UID: "13875841-cfe6-4972-b365-6bd3b23566a4"). InnerVolumeSpecName "kube-api-access-2mg4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:18:50.692737 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.692642 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13875841-cfe6-4972-b365-6bd3b23566a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "13875841-cfe6-4972-b365-6bd3b23566a4" (UID: "13875841-cfe6-4972-b365-6bd3b23566a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:18:50.692843 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.692733 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "13875841-cfe6-4972-b365-6bd3b23566a4" (UID: "13875841-cfe6-4972-b365-6bd3b23566a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:18:50.692843 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.692748 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13875841-cfe6-4972-b365-6bd3b23566a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "13875841-cfe6-4972-b365-6bd3b23566a4" (UID: "13875841-cfe6-4972-b365-6bd3b23566a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 14:18:50.790882 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.790849 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-credential-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:18:50.790882 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.790879 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13875841-cfe6-4972-b365-6bd3b23566a4-istio-podinfo\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:18:50.791108 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.790894 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13875841-cfe6-4972-b365-6bd3b23566a4-istiod-ca-cert\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:18:50.791108 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.790909 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-istio-envoy\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:18:50.791108 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.790921 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13875841-cfe6-4972-b365-6bd3b23566a4-istio-data\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:18:50.791108 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.790933 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mg4j\" (UniqueName: \"kubernetes.io/projected/13875841-cfe6-4972-b365-6bd3b23566a4-kube-api-access-2mg4j\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:18:50.791108 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.790946 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13875841-cfe6-4972-b365-6bd3b23566a4-istio-token\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:18:50.793178 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.793152 2576 generic.go:356] "Generic (PLEG): container finished" podID="13875841-cfe6-4972-b365-6bd3b23566a4" containerID="e49d8e5d4b7a356f4698b93df7ab260f54859369afb905c3adddc17ac5160062" exitCode=0 Jul 10 14:18:50.793289 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.793200 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" Jul 10 14:18:50.793289 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.793232 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" event={"ID":"13875841-cfe6-4972-b365-6bd3b23566a4","Type":"ContainerDied","Data":"e49d8e5d4b7a356f4698b93df7ab260f54859369afb905c3adddc17ac5160062"} Jul 10 14:18:50.793289 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.793283 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps" event={"ID":"13875841-cfe6-4972-b365-6bd3b23566a4","Type":"ContainerDied","Data":"cf0c7f65ff3db31f3baa681c8ea4174f746365d50ca115b21f93ef01d86b53e7"} Jul 10 14:18:50.793398 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.793308 2576 scope.go:117] "RemoveContainer" containerID="e49d8e5d4b7a356f4698b93df7ab260f54859369afb905c3adddc17ac5160062" Jul 10 14:18:50.805166 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.805145 2576 scope.go:117] "RemoveContainer" containerID="f6a2553c4860a889cef7df371f4d7423a32d1495ef4ad7b59f5f08881df1cbeb" Jul 10 14:18:50.813149 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.813129 2576 scope.go:117] "RemoveContainer" containerID="e49d8e5d4b7a356f4698b93df7ab260f54859369afb905c3adddc17ac5160062" Jul 10 14:18:50.813427 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:18:50.813409 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e49d8e5d4b7a356f4698b93df7ab260f54859369afb905c3adddc17ac5160062\": container with ID starting with e49d8e5d4b7a356f4698b93df7ab260f54859369afb905c3adddc17ac5160062 not found: ID does not exist" containerID="e49d8e5d4b7a356f4698b93df7ab260f54859369afb905c3adddc17ac5160062" Jul 10 14:18:50.813466 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.813438 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e49d8e5d4b7a356f4698b93df7ab260f54859369afb905c3adddc17ac5160062"} err="failed to get container status \"e49d8e5d4b7a356f4698b93df7ab260f54859369afb905c3adddc17ac5160062\": rpc error: code = NotFound desc = could not find container \"e49d8e5d4b7a356f4698b93df7ab260f54859369afb905c3adddc17ac5160062\": container with ID starting with e49d8e5d4b7a356f4698b93df7ab260f54859369afb905c3adddc17ac5160062 not found: ID does not exist" Jul 10 14:18:50.813466 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.813455 2576 scope.go:117] "RemoveContainer" containerID="f6a2553c4860a889cef7df371f4d7423a32d1495ef4ad7b59f5f08881df1cbeb" Jul 10 14:18:50.813674 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:18:50.813658 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a2553c4860a889cef7df371f4d7423a32d1495ef4ad7b59f5f08881df1cbeb\": container with ID starting with f6a2553c4860a889cef7df371f4d7423a32d1495ef4ad7b59f5f08881df1cbeb not found: ID does not exist" containerID="f6a2553c4860a889cef7df371f4d7423a32d1495ef4ad7b59f5f08881df1cbeb" Jul 10 14:18:50.813714 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.813679 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a2553c4860a889cef7df371f4d7423a32d1495ef4ad7b59f5f08881df1cbeb"} err="failed to get container status \"f6a2553c4860a889cef7df371f4d7423a32d1495ef4ad7b59f5f08881df1cbeb\": rpc error: code = NotFound desc = could not find container \"f6a2553c4860a889cef7df371f4d7423a32d1495ef4ad7b59f5f08881df1cbeb\": container with ID starting with f6a2553c4860a889cef7df371f4d7423a32d1495ef4ad7b59f5f08881df1cbeb not found: ID does not exist" Jul 10 14:18:50.839833 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.839362 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps"] Jul 10 14:18:50.849338 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:50.849305 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-561ce10f-openshift-default-56d46868b5-wj7ps"] Jul 10 14:18:51.040519 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.040476 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13875841-cfe6-4972-b365-6bd3b23566a4" path="/var/lib/kubelet/pods/13875841-cfe6-4972-b365-6bd3b23566a4/volumes" Jul 10 14:18:51.229441 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.229398 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8"] Jul 10 14:18:51.229965 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.229939 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13875841-cfe6-4972-b365-6bd3b23566a4" containerName="istio-proxy" Jul 10 14:18:51.229965 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.229965 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13875841-cfe6-4972-b365-6bd3b23566a4" containerName="istio-proxy" Jul 10 14:18:51.230158 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.229992 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13875841-cfe6-4972-b365-6bd3b23566a4" containerName="istio-proxy" Jul 10 14:18:51.230158 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.230001 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13875841-cfe6-4972-b365-6bd3b23566a4" containerName="istio-proxy" Jul 10 14:18:51.230158 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.230102 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="13875841-cfe6-4972-b365-6bd3b23566a4" containerName="istio-proxy" Jul 10 14:18:51.235240 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.235171 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8" Jul 10 14:18:51.239286 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.239266 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8qd8l\"" Jul 10 14:18:51.239589 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.239423 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 14:18:51.256953 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.256907 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8"] Jul 10 14:18:51.295632 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.295541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8\" (UID: \"ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8" Jul 10 14:18:51.396762 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.396719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8\" (UID: \"ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8" Jul 10 14:18:51.397335 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.397314 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8\" (UID: \"ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8" Jul 10 14:18:51.550483 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.550368 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8" Jul 10 14:18:51.714412 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.714387 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8"] Jul 10 14:18:51.716865 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:18:51.716830 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8a9dce_8a3e_4fb2_a1d7_685b667d0d06.slice/crio-870b94996d12baca6a62d22ade67aac86104a293d1b7764e26a524f22b9043e5 WatchSource:0}: Error finding container 870b94996d12baca6a62d22ade67aac86104a293d1b7764e26a524f22b9043e5: Status 404 returned error can't find the container with id 870b94996d12baca6a62d22ade67aac86104a293d1b7764e26a524f22b9043e5 Jul 10 14:18:51.740370 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.740231 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:18:51.809423 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.809334 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8" event={"ID":"ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06","Type":"ContainerStarted","Data":"870b94996d12baca6a62d22ade67aac86104a293d1b7764e26a524f22b9043e5"} Jul 10 14:18:51.841512 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:51.841467 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:18:52.816138 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:52.816099 2576 generic.go:356] "Generic (PLEG): container finished" podID="ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06" containerID="525fdb5b402decc19097aed2be46e23305f0fea79d3643d76682c56b91ca18ed" exitCode=7 Jul 10 14:18:52.816534 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:52.816188 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8" event={"ID":"ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06","Type":"ContainerDied","Data":"525fdb5b402decc19097aed2be46e23305f0fea79d3643d76682c56b91ca18ed"} Jul 10 14:18:52.816534 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:52.816352 2576 scope.go:117] "RemoveContainer" containerID="525fdb5b402decc19097aed2be46e23305f0fea79d3643d76682c56b91ca18ed" Jul 10 14:18:53.822376 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:53.822335 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8" event={"ID":"ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06","Type":"ContainerStarted","Data":"e1920fda314a751b37293c49659cf0110f560331a7b13aca382daefb104e2cf6"} Jul 10 14:18:53.853840 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:53.853729 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8" podStartSLOduration=2.853710984 podStartE2EDuration="2.853710984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:18:53.851995682 +0000 UTC m=+1741.315601132" watchObservedRunningTime="2026-07-10 14:18:53.853710984 +0000 UTC m=+1741.317316430" Jul 10 14:18:54.829885 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:54.829769 2576 generic.go:356] "Generic (PLEG): container finished" podID="ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06" containerID="e1920fda314a751b37293c49659cf0110f560331a7b13aca382daefb104e2cf6" exitCode=7 Jul 10 14:18:54.829885 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:54.829860 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8" event={"ID":"ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06","Type":"ContainerDied","Data":"e1920fda314a751b37293c49659cf0110f560331a7b13aca382daefb104e2cf6"} Jul 10 14:18:54.830371 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:54.829932 2576 scope.go:117] "RemoveContainer" containerID="525fdb5b402decc19097aed2be46e23305f0fea79d3643d76682c56b91ca18ed" Jul 10 14:18:54.830371 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:54.830194 2576 scope.go:117] "RemoveContainer" containerID="e1920fda314a751b37293c49659cf0110f560331a7b13aca382daefb104e2cf6" Jul 10 14:18:54.830481 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:18:54.830437 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8_odh-ai-gateway-infra(ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8" podUID="ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06" Jul 10 14:18:55.178067 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:55.178024 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:18:55.836026 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:55.835999 2576 scope.go:117] "RemoveContainer" containerID="e1920fda314a751b37293c49659cf0110f560331a7b13aca382daefb104e2cf6" Jul 10 14:18:55.836399 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:18:55.836187 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8_odh-ai-gateway-infra(ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8" podUID="ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06" Jul 10 14:18:57.846446 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:57.846414 2576 generic.go:356] "Generic (PLEG): container finished" podID="2db3ea0c-a225-42fd-ac1c-a431e6b61d59" containerID="1e0420dd1331a6028fa672a5fab8b2f2fb8ff80269118deeadcff6945ae9669b" exitCode=0 Jul 10 14:18:57.846943 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:57.846479 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" event={"ID":"2db3ea0c-a225-42fd-ac1c-a431e6b61d59","Type":"ContainerDied","Data":"1e0420dd1331a6028fa672a5fab8b2f2fb8ff80269118deeadcff6945ae9669b"} Jul 10 14:18:57.846943 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:57.846878 2576 scope.go:117] "RemoveContainer" containerID="1e0420dd1331a6028fa672a5fab8b2f2fb8ff80269118deeadcff6945ae9669b" Jul 10 14:18:57.847876 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:57.847849 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:18:57.848127 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:57.847917 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:18:57.848127 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:57.847967 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:18:58.854278 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:58.854243 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" event={"ID":"2db3ea0c-a225-42fd-ac1c-a431e6b61d59","Type":"ContainerStarted","Data":"9577cc7215d2c5c31472af7cd3aa85b62650e3424ae16c4066b585a4a10da268"} Jul 10 14:18:58.968566 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:58.968536 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:18:58.969554 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:58.969537 2576 patch_prober.go:28] interesting pod/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 10 14:18:58.969630 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:58.969575 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" podUID="2db3ea0c-a225-42fd-ac1c-a431e6b61d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 10 14:18:59.968613 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:59.968578 2576 patch_prober.go:28] interesting pod/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 10 14:18:59.969128 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:18:59.968639 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" podUID="2db3ea0c-a225-42fd-ac1c-a431e6b61d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 10 14:19:00.968835 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:00.968768 2576 patch_prober.go:28] interesting pod/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 10 14:19:00.969332 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:00.968879 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" podUID="2db3ea0c-a225-42fd-ac1c-a431e6b61d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 10 14:19:01.969483 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:01.969444 2576 patch_prober.go:28] interesting pod/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 10 14:19:01.969949 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:01.969498 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" podUID="2db3ea0c-a225-42fd-ac1c-a431e6b61d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 10 14:19:02.972946 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:02.972911 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:19:02.973361 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:02.973289 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:19:02.973905 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:02.973888 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:19:11.031010 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:11.030973 2576 scope.go:117] "RemoveContainer" containerID="e1920fda314a751b37293c49659cf0110f560331a7b13aca382daefb104e2cf6" Jul 10 14:19:11.918276 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:11.918244 2576 generic.go:356] "Generic (PLEG): container finished" podID="ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06" containerID="a77ca2100177a9282dbd7000c9b33d4e24dcf118c484fa1e9b59e55fafea7219" exitCode=0 Jul 10 14:19:11.918440 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:11.918313 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8" event={"ID":"ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06","Type":"ContainerDied","Data":"a77ca2100177a9282dbd7000c9b33d4e24dcf118c484fa1e9b59e55fafea7219"} Jul 10 14:19:11.918440 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:11.918371 2576 scope.go:117] "RemoveContainer" containerID="e1920fda314a751b37293c49659cf0110f560331a7b13aca382daefb104e2cf6" Jul 10 14:19:12.963188 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:12.963154 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8"] Jul 10 14:19:13.058974 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:13.058943 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8" Jul 10 14:19:13.202668 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:13.202551 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06-maas-api-service-ca\") pod \"ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06\" (UID: \"ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06\") " Jul 10 14:19:13.203014 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:13.202986 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06" (UID: "ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:19:13.303989 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:13.303957 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06-maas-api-service-ca\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:19:13.930208 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:13.930179 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8" Jul 10 14:19:13.930430 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:13.930209 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8" event={"ID":"ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06","Type":"ContainerDied","Data":"870b94996d12baca6a62d22ade67aac86104a293d1b7764e26a524f22b9043e5"} Jul 10 14:19:13.930430 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:13.930253 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="870b94996d12baca6a62d22ade67aac86104a293d1b7764e26a524f22b9043e5" Jul 10 14:19:13.962700 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:13.962663 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8"] Jul 10 14:19:13.965264 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:13.965240 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-s4jb8"] Jul 10 14:19:15.036099 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:15.036071 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06" path="/var/lib/kubelet/pods/ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06/volumes" Jul 10 14:19:16.235869 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:16.235832 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:19:17.858436 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:17.858402 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:19:21.935978 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:21.935946 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:19:22.033464 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:22.033433 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:19:51.271471 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:51.271431 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5"] Jul 10 14:19:51.272028 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:51.271959 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06" containerName="revoke-keys" Jul 10 14:19:51.272028 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:51.271980 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06" containerName="revoke-keys" Jul 10 14:19:51.272028 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:51.271995 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06" containerName="revoke-keys" Jul 10 14:19:51.272028 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:51.272004 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06" containerName="revoke-keys" Jul 10 14:19:51.272238 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:51.272100 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06" containerName="revoke-keys" Jul 10 14:19:51.272238 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:51.272113 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06" containerName="revoke-keys" Jul 10 14:19:51.272238 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:51.272125 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="13875841-cfe6-4972-b365-6bd3b23566a4" containerName="istio-proxy" Jul 10 14:19:51.272238 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:51.272134 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06" containerName="revoke-keys" Jul 10 14:19:51.275208 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:51.275181 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5" Jul 10 14:19:51.277845 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:51.277799 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 14:19:51.277966 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:51.277872 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8qd8l\"" Jul 10 14:19:51.282879 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:51.282847 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5"] Jul 10 14:19:51.343707 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:51.343663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e7a6c2-f4e0-424a-91c8-b702e31983f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5\" (UID: \"54e7a6c2-f4e0-424a-91c8-b702e31983f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5" Jul 10 14:19:51.445262 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:51.445163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e7a6c2-f4e0-424a-91c8-b702e31983f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5\" (UID: \"54e7a6c2-f4e0-424a-91c8-b702e31983f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5" Jul 10 14:19:51.445856 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:51.445798 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e7a6c2-f4e0-424a-91c8-b702e31983f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5\" (UID: \"54e7a6c2-f4e0-424a-91c8-b702e31983f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5" Jul 10 14:19:51.589566 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:51.589524 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5" Jul 10 14:19:51.727871 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:51.726621 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5"] Jul 10 14:19:51.730725 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:19:51.730685 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e7a6c2_f4e0_424a_91c8_b702e31983f1.slice/crio-08a766fae2ad705e26785460049d84ec8c14d79858b0541da61d45e4b25085ba WatchSource:0}: Error finding container 08a766fae2ad705e26785460049d84ec8c14d79858b0541da61d45e4b25085ba: Status 404 returned error can't find the container with id 08a766fae2ad705e26785460049d84ec8c14d79858b0541da61d45e4b25085ba Jul 10 14:19:52.115319 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:52.115282 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5" event={"ID":"54e7a6c2-f4e0-424a-91c8-b702e31983f1","Type":"ContainerStarted","Data":"bd3ab71f56bb221e5bd2fdf860225718acfc40ed4f0e8ab150178c3ea7176d64"} Jul 10 14:19:52.115509 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:52.115338 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5" event={"ID":"54e7a6c2-f4e0-424a-91c8-b702e31983f1","Type":"ContainerStarted","Data":"08a766fae2ad705e26785460049d84ec8c14d79858b0541da61d45e4b25085ba"} Jul 10 14:19:52.141982 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:52.141923 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5" podStartSLOduration=1.141905789 podStartE2EDuration="1.141905789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:19:52.139902568 +0000 UTC m=+1799.603508015" watchObservedRunningTime="2026-07-10 14:19:52.141905789 +0000 UTC m=+1799.605511236" Jul 10 14:19:53.071232 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:53.071202 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 14:19:53.073291 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:53.073272 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 14:19:53.121246 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:53.121208 2576 generic.go:356] "Generic (PLEG): container finished" podID="54e7a6c2-f4e0-424a-91c8-b702e31983f1" containerID="bd3ab71f56bb221e5bd2fdf860225718acfc40ed4f0e8ab150178c3ea7176d64" exitCode=0 Jul 10 14:19:53.121422 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:53.121297 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5" event={"ID":"54e7a6c2-f4e0-424a-91c8-b702e31983f1","Type":"ContainerDied","Data":"bd3ab71f56bb221e5bd2fdf860225718acfc40ed4f0e8ab150178c3ea7176d64"} Jul 10 14:19:54.270225 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:54.270196 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5" Jul 10 14:19:54.370700 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:54.370663 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e7a6c2-f4e0-424a-91c8-b702e31983f1-maas-api-service-ca\") pod \"54e7a6c2-f4e0-424a-91c8-b702e31983f1\" (UID: \"54e7a6c2-f4e0-424a-91c8-b702e31983f1\") " Jul 10 14:19:54.371084 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:54.371061 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e7a6c2-f4e0-424a-91c8-b702e31983f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "54e7a6c2-f4e0-424a-91c8-b702e31983f1" (UID: "54e7a6c2-f4e0-424a-91c8-b702e31983f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:19:54.472289 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:54.472243 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e7a6c2-f4e0-424a-91c8-b702e31983f1-maas-api-service-ca\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:19:55.131393 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:55.131359 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5" event={"ID":"54e7a6c2-f4e0-424a-91c8-b702e31983f1","Type":"ContainerDied","Data":"08a766fae2ad705e26785460049d84ec8c14d79858b0541da61d45e4b25085ba"} Jul 10 14:19:55.131565 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:55.131408 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08a766fae2ad705e26785460049d84ec8c14d79858b0541da61d45e4b25085ba" Jul 10 14:19:55.131565 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:19:55.131377 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5" Jul 10 14:20:01.922953 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:01.922916 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:04.800587 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:04.800556 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:20:06.371782 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:06.371742 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:12.891008 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:12.890971 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:12.992692 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:12.992662 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:16.554547 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:16.554504 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557"] Jul 10 14:20:16.555062 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:16.554797 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" podUID="2db3ea0c-a225-42fd-ac1c-a431e6b61d59" containerName="istio-proxy" containerID="cri-o://9577cc7215d2c5c31472af7cd3aa85b62650e3424ae16c4066b585a4a10da268" gracePeriod=30 Jul 10 14:20:16.586227 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:16.586192 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:17.194247 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:17.194201 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:17.292036 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:17.292003 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:17.391107 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:17.391076 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:17.975884 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:17.975846 2576 patch_prober.go:28] interesting pod/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 14:20:17.976352 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:17.975923 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" podUID="2db3ea0c-a225-42fd-ac1c-a431e6b61d59" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 14:20:18.897305 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:18.897265 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:21.799427 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.799395 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:20:21.816556 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.816465 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-podinfo\") pod \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " Jul 10 14:20:21.816556 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.816520 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-workload-certs\") pod \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " Jul 10 14:20:21.816556 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.816558 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-credential-socket\") pod \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " Jul 10 14:20:21.816895 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.816725 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-workload-socket\") pod \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " Jul 10 14:20:21.816895 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.816769 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-envoy\") pod \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " Jul 10 14:20:21.816895 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.816821 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2db3ea0c-a225-42fd-ac1c-a431e6b61d59" (UID: "2db3ea0c-a225-42fd-ac1c-a431e6b61d59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:20:21.816895 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.816842 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfktb\" (UniqueName: \"kubernetes.io/projected/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-kube-api-access-rfktb\") pod \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " Jul 10 14:20:21.816895 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.816847 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2db3ea0c-a225-42fd-ac1c-a431e6b61d59" (UID: "2db3ea0c-a225-42fd-ac1c-a431e6b61d59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:20:21.816895 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.816893 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-token\") pod \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " Jul 10 14:20:21.817178 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.816920 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-data\") pod \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " Jul 10 14:20:21.817178 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.816979 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istiod-ca-cert\") pod \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\" (UID: \"2db3ea0c-a225-42fd-ac1c-a431e6b61d59\") " Jul 10 14:20:21.817178 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.816998 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2db3ea0c-a225-42fd-ac1c-a431e6b61d59" (UID: "2db3ea0c-a225-42fd-ac1c-a431e6b61d59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:20:21.817345 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.817269 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-workload-certs\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:20:21.817345 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.817287 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-credential-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:20:21.817345 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.817302 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-workload-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:20:21.817489 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.817417 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2db3ea0c-a225-42fd-ac1c-a431e6b61d59" (UID: "2db3ea0c-a225-42fd-ac1c-a431e6b61d59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:20:21.818020 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.817995 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-data" (OuterVolumeSpecName: "istio-data") pod "2db3ea0c-a225-42fd-ac1c-a431e6b61d59" (UID: "2db3ea0c-a225-42fd-ac1c-a431e6b61d59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:20:21.819062 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.819038 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2db3ea0c-a225-42fd-ac1c-a431e6b61d59" (UID: "2db3ea0c-a225-42fd-ac1c-a431e6b61d59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:20:21.819126 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.819043 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2db3ea0c-a225-42fd-ac1c-a431e6b61d59" (UID: "2db3ea0c-a225-42fd-ac1c-a431e6b61d59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 14:20:21.819331 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.819308 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-token" (OuterVolumeSpecName: "istio-token") pod "2db3ea0c-a225-42fd-ac1c-a431e6b61d59" (UID: "2db3ea0c-a225-42fd-ac1c-a431e6b61d59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:20:21.819387 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.819323 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-kube-api-access-rfktb" (OuterVolumeSpecName: "kube-api-access-rfktb") pod "2db3ea0c-a225-42fd-ac1c-a431e6b61d59" (UID: "2db3ea0c-a225-42fd-ac1c-a431e6b61d59"). InnerVolumeSpecName "kube-api-access-rfktb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:20:21.918509 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.918465 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-podinfo\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:20:21.918509 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.918498 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-envoy\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:20:21.918509 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.918508 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfktb\" (UniqueName: \"kubernetes.io/projected/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-kube-api-access-rfktb\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:20:21.918509 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.918517 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-token\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:20:21.918789 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.918526 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istio-data\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:20:21.918789 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:21.918533 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db3ea0c-a225-42fd-ac1c-a431e6b61d59-istiod-ca-cert\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:20:22.264450 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:22.264419 2576 generic.go:356] "Generic (PLEG): container finished" podID="2db3ea0c-a225-42fd-ac1c-a431e6b61d59" containerID="9577cc7215d2c5c31472af7cd3aa85b62650e3424ae16c4066b585a4a10da268" exitCode=0 Jul 10 14:20:22.264832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:22.264473 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" Jul 10 14:20:22.264832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:22.264510 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" event={"ID":"2db3ea0c-a225-42fd-ac1c-a431e6b61d59","Type":"ContainerDied","Data":"9577cc7215d2c5c31472af7cd3aa85b62650e3424ae16c4066b585a4a10da268"} Jul 10 14:20:22.264832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:22.264566 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557" event={"ID":"2db3ea0c-a225-42fd-ac1c-a431e6b61d59","Type":"ContainerDied","Data":"f9d0d17899e703509be1ff15f18c5fed88df7ad01337fe1fe67ff85586a279be"} Jul 10 14:20:22.264832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:22.264589 2576 scope.go:117] "RemoveContainer" containerID="9577cc7215d2c5c31472af7cd3aa85b62650e3424ae16c4066b585a4a10da268" Jul 10 14:20:22.274844 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:22.274803 2576 scope.go:117] "RemoveContainer" containerID="1e0420dd1331a6028fa672a5fab8b2f2fb8ff80269118deeadcff6945ae9669b" Jul 10 14:20:22.283728 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:22.283702 2576 scope.go:117] "RemoveContainer" containerID="9577cc7215d2c5c31472af7cd3aa85b62650e3424ae16c4066b585a4a10da268" Jul 10 14:20:22.284082 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:20:22.284057 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9577cc7215d2c5c31472af7cd3aa85b62650e3424ae16c4066b585a4a10da268\": container with ID starting with 9577cc7215d2c5c31472af7cd3aa85b62650e3424ae16c4066b585a4a10da268 not found: ID does not exist" containerID="9577cc7215d2c5c31472af7cd3aa85b62650e3424ae16c4066b585a4a10da268" Jul 10 14:20:22.284147 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:22.284097 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9577cc7215d2c5c31472af7cd3aa85b62650e3424ae16c4066b585a4a10da268"} err="failed to get container status \"9577cc7215d2c5c31472af7cd3aa85b62650e3424ae16c4066b585a4a10da268\": rpc error: code = NotFound desc = could not find container \"9577cc7215d2c5c31472af7cd3aa85b62650e3424ae16c4066b585a4a10da268\": container with ID starting with 9577cc7215d2c5c31472af7cd3aa85b62650e3424ae16c4066b585a4a10da268 not found: ID does not exist" Jul 10 14:20:22.284147 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:22.284123 2576 scope.go:117] "RemoveContainer" containerID="1e0420dd1331a6028fa672a5fab8b2f2fb8ff80269118deeadcff6945ae9669b" Jul 10 14:20:22.284429 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:20:22.284405 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e0420dd1331a6028fa672a5fab8b2f2fb8ff80269118deeadcff6945ae9669b\": container with ID starting with 1e0420dd1331a6028fa672a5fab8b2f2fb8ff80269118deeadcff6945ae9669b not found: ID does not exist" containerID="1e0420dd1331a6028fa672a5fab8b2f2fb8ff80269118deeadcff6945ae9669b" Jul 10 14:20:22.284549 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:22.284431 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e0420dd1331a6028fa672a5fab8b2f2fb8ff80269118deeadcff6945ae9669b"} err="failed to get container status \"1e0420dd1331a6028fa672a5fab8b2f2fb8ff80269118deeadcff6945ae9669b\": rpc error: code = NotFound desc = could not find container \"1e0420dd1331a6028fa672a5fab8b2f2fb8ff80269118deeadcff6945ae9669b\": container with ID starting with 1e0420dd1331a6028fa672a5fab8b2f2fb8ff80269118deeadcff6945ae9669b not found: ID does not exist" Jul 10 14:20:22.290863 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:22.290834 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557"] Jul 10 14:20:22.298362 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:22.298331 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-cce0783f-openshift-default-7bdbddbb59-f7557"] Jul 10 14:20:23.037206 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:23.037173 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2db3ea0c-a225-42fd-ac1c-a431e6b61d59" path="/var/lib/kubelet/pods/2db3ea0c-a225-42fd-ac1c-a431e6b61d59/volumes" Jul 10 14:20:25.759799 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:25.759761 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:25.862090 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:25.862044 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:25.960617 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:25.960576 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:26.060026 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:26.059942 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:26.499294 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:26.499263 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:28.880841 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:28.880777 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:28.949030 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:28.948995 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:29.207543 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:29.207504 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:45.996887 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:45.996852 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:52.589004 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:52.588971 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:59.765402 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:59.765360 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:59.849661 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:59.849629 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:20:59.971528 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:20:59.971490 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:00.078348 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:00.078257 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:16.092102 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:16.092010 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:22.857135 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:22.857093 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:22.939670 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:22.939634 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:23.063863 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:23.063833 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:33.637935 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:33.637893 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:34.514055 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:34.514026 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:41.221963 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:41.221927 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:41.289019 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:41.288981 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:41.385440 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:41.385402 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:41.485443 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:41.485356 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:41.591107 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:41.591072 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:42.418010 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:42.417968 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:42.506899 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:42.506865 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:42.608889 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:42.608855 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:44.017862 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:44.017822 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:21:44.347735 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:44.347641 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:51.377084 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:51.377037 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:51.476502 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:51.476466 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:51.795203 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:51.795160 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:21:52.158132 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:52.158101 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:59.013066 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.013017 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:59.111154 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.111120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:59.253197 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.253163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:59.307533 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.307434 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:21:59.617630 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.617542 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8"] Jul 10 14:21:59.617890 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.617877 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db3ea0c-a225-42fd-ac1c-a431e6b61d59" containerName="istio-proxy" Jul 10 14:21:59.617940 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.617892 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db3ea0c-a225-42fd-ac1c-a431e6b61d59" containerName="istio-proxy" Jul 10 14:21:59.617940 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.617911 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54e7a6c2-f4e0-424a-91c8-b702e31983f1" containerName="revoke-keys" Jul 10 14:21:59.617940 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.617917 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="54e7a6c2-f4e0-424a-91c8-b702e31983f1" containerName="revoke-keys" Jul 10 14:21:59.617940 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.617931 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06" containerName="revoke-keys" Jul 10 14:21:59.617940 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.617936 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8a9dce-8a3e-4fb2-a1d7-685b667d0d06" containerName="revoke-keys" Jul 10 14:21:59.618094 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.617987 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2db3ea0c-a225-42fd-ac1c-a431e6b61d59" containerName="istio-proxy" Jul 10 14:21:59.618094 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.617997 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2db3ea0c-a225-42fd-ac1c-a431e6b61d59" containerName="istio-proxy" Jul 10 14:21:59.618094 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.618003 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="54e7a6c2-f4e0-424a-91c8-b702e31983f1" containerName="revoke-keys" Jul 10 14:21:59.620820 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.620786 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.627653 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.627432 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a6e9e1cb-openshift-default-dockercfg-kgdxg\"" Jul 10 14:21:59.632788 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.632766 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:59.656756 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.656726 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8"] Jul 10 14:21:59.765686 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.765650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-workload-certs\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.765905 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.765695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-data\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.765905 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.765720 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-workload-socket\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.765905 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.765759 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-credential-socket\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.765905 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.765797 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istiod-ca-cert\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.765905 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.765873 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-podinfo\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.765905 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.765900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-envoy\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.766117 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.765926 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2mkw\" (UniqueName: \"kubernetes.io/projected/74cc2ac9-6312-46c1-9559-b99ca5c95a60-kube-api-access-s2mkw\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.766117 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.765946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-token\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.773960 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.773928 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:21:59.867160 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.867117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-workload-certs\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.867160 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.867164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-data\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.867423 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.867185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-workload-socket\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.867423 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.867205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-credential-socket\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.867423 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.867238 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istiod-ca-cert\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.867423 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.867268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-podinfo\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.867423 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.867295 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-envoy\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.867423 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.867316 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2mkw\" (UniqueName: \"kubernetes.io/projected/74cc2ac9-6312-46c1-9559-b99ca5c95a60-kube-api-access-s2mkw\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.867423 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.867411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-token\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.867856 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.867639 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-credential-socket\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.867856 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.867671 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-data\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.867856 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.867730 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-workload-socket\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.868050 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.867907 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-workload-certs\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.868108 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.868082 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istiod-ca-cert\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.869659 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.869637 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-envoy\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.869952 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.869934 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-podinfo\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.877201 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.877176 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-token\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.877346 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.877328 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2mkw\" (UniqueName: \"kubernetes.io/projected/74cc2ac9-6312-46c1-9559-b99ca5c95a60-kube-api-access-s2mkw\") pod \"e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:21:59.931499 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:21:59.931459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:22:00.080281 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:00.074802 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8"] Jul 10 14:22:00.082166 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:00.082123 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:22:00.082272 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:00.082205 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:22:00.082272 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:00.082247 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:22:00.738275 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:00.738233 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" event={"ID":"74cc2ac9-6312-46c1-9559-b99ca5c95a60","Type":"ContainerStarted","Data":"6d56adf284f34794250e40ed4de979205629049e4da565cd078c7aaa30d0593d"} Jul 10 14:22:00.738460 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:00.738292 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" event={"ID":"74cc2ac9-6312-46c1-9559-b99ca5c95a60","Type":"ContainerStarted","Data":"2f0d1c06c7f094e681ba9da6c03aa2fddcb911e45ad0a9c799763b0c8045a65c"} Jul 10 14:22:00.778115 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:00.778060 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" podStartSLOduration=1.778044034 podStartE2EDuration="1.778044034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:22:00.777791901 +0000 UTC m=+1928.241397348" watchObservedRunningTime="2026-07-10 14:22:00.778044034 +0000 UTC m=+1928.241649479" Jul 10 14:22:00.931949 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:00.931896 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:22:01.937169 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:01.937139 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:22:02.748952 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:02.748920 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:22:02.749988 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:02.749967 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:22:02.847071 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:02.847039 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:02.941482 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:02.941452 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:03.044906 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:03.044785 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:03.414625 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:03.414594 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:03.678372 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:03.678279 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:03.774919 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:03.774880 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:07.374766 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:07.374726 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:07.603943 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:07.603901 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:22:08.085611 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:08.085578 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:08.236904 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:08.236869 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:08.547599 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:08.547561 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:15.826595 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:15.826490 2576 generic.go:356] "Generic (PLEG): container finished" podID="74cc2ac9-6312-46c1-9559-b99ca5c95a60" containerID="6d56adf284f34794250e40ed4de979205629049e4da565cd078c7aaa30d0593d" exitCode=0 Jul 10 14:22:15.826595 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:15.826543 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" event={"ID":"74cc2ac9-6312-46c1-9559-b99ca5c95a60","Type":"ContainerDied","Data":"6d56adf284f34794250e40ed4de979205629049e4da565cd078c7aaa30d0593d"} Jul 10 14:22:15.827071 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:15.826941 2576 scope.go:117] "RemoveContainer" containerID="6d56adf284f34794250e40ed4de979205629049e4da565cd078c7aaa30d0593d" Jul 10 14:22:15.827980 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:15.827947 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:22:15.828102 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:15.828035 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:22:15.828102 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:15.828084 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:22:15.865396 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:15.865214 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:15.986398 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:15.986366 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:16.239386 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:16.239346 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6e9e1cb-hc6l2"] Jul 10 14:22:16.239747 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:16.239730 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db3ea0c-a225-42fd-ac1c-a431e6b61d59" containerName="istio-proxy" Jul 10 14:22:16.239868 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:16.239750 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db3ea0c-a225-42fd-ac1c-a431e6b61d59" containerName="istio-proxy" Jul 10 14:22:16.241951 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:16.241931 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6e9e1cb-hc6l2" Jul 10 14:22:16.245085 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:16.245067 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 14:22:16.245836 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:16.245820 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8qd8l\"" Jul 10 14:22:16.274727 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:16.274671 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6e9e1cb-hc6l2"] Jul 10 14:22:16.317084 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:16.317041 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a484a423-ef61-4798-bf44-a0cb98767f14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a6e9e1cb-hc6l2\" (UID: \"a484a423-ef61-4798-bf44-a0cb98767f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6e9e1cb-hc6l2" Jul 10 14:22:16.418649 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:16.418609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a484a423-ef61-4798-bf44-a0cb98767f14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a6e9e1cb-hc6l2\" (UID: \"a484a423-ef61-4798-bf44-a0cb98767f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6e9e1cb-hc6l2" Jul 10 14:22:16.419140 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:16.419119 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a484a423-ef61-4798-bf44-a0cb98767f14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a6e9e1cb-hc6l2\" (UID: \"a484a423-ef61-4798-bf44-a0cb98767f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6e9e1cb-hc6l2" Jul 10 14:22:16.552597 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:16.552507 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6e9e1cb-hc6l2" Jul 10 14:22:16.703898 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:16.700170 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6e9e1cb-hc6l2"] Jul 10 14:22:16.705518 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:22:16.705487 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda484a423_ef61_4798_bf44_a0cb98767f14.slice/crio-ec175494a0537a74f381f3f28af2cae3cd1f66a92e624b9bf20387d0b259bab3 WatchSource:0}: Error finding container ec175494a0537a74f381f3f28af2cae3cd1f66a92e624b9bf20387d0b259bab3: Status 404 returned error can't find the container with id ec175494a0537a74f381f3f28af2cae3cd1f66a92e624b9bf20387d0b259bab3 Jul 10 14:22:16.836916 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:16.836781 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" event={"ID":"74cc2ac9-6312-46c1-9559-b99ca5c95a60","Type":"ContainerStarted","Data":"5c57d5b595f1d8488eb6e3d51d17fd1e2c5111f0f42b5b6815864d45ffbf710d"} Jul 10 14:22:16.839706 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:16.839662 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6e9e1cb-hc6l2" event={"ID":"a484a423-ef61-4798-bf44-a0cb98767f14","Type":"ContainerStarted","Data":"ec175494a0537a74f381f3f28af2cae3cd1f66a92e624b9bf20387d0b259bab3"} Jul 10 14:22:16.932695 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:16.932657 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:22:16.933776 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:16.933742 2576 patch_prober.go:28] interesting pod/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 10 14:22:16.933935 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:16.933820 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" podUID="74cc2ac9-6312-46c1-9559-b99ca5c95a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 10 14:22:17.749366 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:17.749277 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:22:17.848908 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:17.848868 2576 generic.go:356] "Generic (PLEG): container finished" podID="a484a423-ef61-4798-bf44-a0cb98767f14" containerID="51623448989c6f2e30d933281924c9f6be97a3a2e40e674c634d45d0fcb695d5" exitCode=0 Jul 10 14:22:17.849306 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:17.848982 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6e9e1cb-hc6l2" event={"ID":"a484a423-ef61-4798-bf44-a0cb98767f14","Type":"ContainerDied","Data":"51623448989c6f2e30d933281924c9f6be97a3a2e40e674c634d45d0fcb695d5"} Jul 10 14:22:17.933271 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:17.932787 2576 patch_prober.go:28] interesting pod/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 10 14:22:17.933271 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:17.932882 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" podUID="74cc2ac9-6312-46c1-9559-b99ca5c95a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 10 14:22:18.941543 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:18.940944 2576 patch_prober.go:28] interesting pod/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 10 14:22:18.941543 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:18.941018 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" podUID="74cc2ac9-6312-46c1-9559-b99ca5c95a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 10 14:22:19.001112 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:19.001085 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6e9e1cb-hc6l2" Jul 10 14:22:19.152441 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:19.152401 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a484a423-ef61-4798-bf44-a0cb98767f14-maas-api-service-ca\") pod \"a484a423-ef61-4798-bf44-a0cb98767f14\" (UID: \"a484a423-ef61-4798-bf44-a0cb98767f14\") " Jul 10 14:22:19.152871 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:19.152846 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a484a423-ef61-4798-bf44-a0cb98767f14-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a484a423-ef61-4798-bf44-a0cb98767f14" (UID: "a484a423-ef61-4798-bf44-a0cb98767f14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:22:19.253430 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:19.253385 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a484a423-ef61-4798-bf44-a0cb98767f14-maas-api-service-ca\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:22:19.860457 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:19.860428 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6e9e1cb-hc6l2" Jul 10 14:22:19.860674 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:19.860468 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6e9e1cb-hc6l2" event={"ID":"a484a423-ef61-4798-bf44-a0cb98767f14","Type":"ContainerDied","Data":"ec175494a0537a74f381f3f28af2cae3cd1f66a92e624b9bf20387d0b259bab3"} Jul 10 14:22:19.860674 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:19.860516 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec175494a0537a74f381f3f28af2cae3cd1f66a92e624b9bf20387d0b259bab3" Jul 10 14:22:19.932495 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:19.932033 2576 patch_prober.go:28] interesting pod/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 10 14:22:19.932495 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:19.932101 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" podUID="74cc2ac9-6312-46c1-9559-b99ca5c95a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 10 14:22:20.932095 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:20.932051 2576 patch_prober.go:28] interesting pod/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 10 14:22:20.932496 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:20.932127 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" podUID="74cc2ac9-6312-46c1-9559-b99ca5c95a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 10 14:22:21.935964 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:21.935937 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:22:21.936998 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:21.936977 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:22:27.039046 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:27.039007 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:30.032820 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:30.032771 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:22:31.368605 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:31.368567 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:39.132396 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:39.132352 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:39.294067 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:39.294028 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:42.068621 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:42.068585 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8"] Jul 10 14:22:42.069064 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:42.068839 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" podUID="74cc2ac9-6312-46c1-9559-b99ca5c95a60" containerName="istio-proxy" containerID="cri-o://5c57d5b595f1d8488eb6e3d51d17fd1e2c5111f0f42b5b6815864d45ffbf710d" gracePeriod=30 Jul 10 14:22:42.090464 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:42.090435 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:42.320626 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:42.320542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:42.469673 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:42.469634 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:42.670590 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:42.670558 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:42.785261 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:42.785223 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:47.323055 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.323021 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:22:47.392347 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.392316 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2mkw\" (UniqueName: \"kubernetes.io/projected/74cc2ac9-6312-46c1-9559-b99ca5c95a60-kube-api-access-s2mkw\") pod \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " Jul 10 14:22:47.392542 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.392358 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istiod-ca-cert\") pod \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " Jul 10 14:22:47.392542 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.392391 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-podinfo\") pod \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " Jul 10 14:22:47.392688 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.392534 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-data\") pod \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " Jul 10 14:22:47.392688 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.392575 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-token\") pod \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " Jul 10 14:22:47.392688 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.392612 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-workload-socket\") pod \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " Jul 10 14:22:47.392688 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.392644 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-credential-socket\") pod \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " Jul 10 14:22:47.392688 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.392683 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-envoy\") pod \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " Jul 10 14:22:47.392988 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.392721 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-workload-certs\") pod \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\" (UID: \"74cc2ac9-6312-46c1-9559-b99ca5c95a60\") " Jul 10 14:22:47.392988 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.392916 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "74cc2ac9-6312-46c1-9559-b99ca5c95a60" (UID: "74cc2ac9-6312-46c1-9559-b99ca5c95a60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:22:47.392988 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.392947 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "74cc2ac9-6312-46c1-9559-b99ca5c95a60" (UID: "74cc2ac9-6312-46c1-9559-b99ca5c95a60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:22:47.393152 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.393008 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "74cc2ac9-6312-46c1-9559-b99ca5c95a60" (UID: "74cc2ac9-6312-46c1-9559-b99ca5c95a60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:22:47.393152 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.393140 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "74cc2ac9-6312-46c1-9559-b99ca5c95a60" (UID: "74cc2ac9-6312-46c1-9559-b99ca5c95a60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:22:47.393265 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.393241 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-workload-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:22:47.393308 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.393262 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-credential-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:22:47.393308 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.393277 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istiod-ca-cert\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:22:47.393558 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.393536 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-data" (OuterVolumeSpecName: "istio-data") pod "74cc2ac9-6312-46c1-9559-b99ca5c95a60" (UID: "74cc2ac9-6312-46c1-9559-b99ca5c95a60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:22:47.395132 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.395097 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "74cc2ac9-6312-46c1-9559-b99ca5c95a60" (UID: "74cc2ac9-6312-46c1-9559-b99ca5c95a60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 14:22:47.395242 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.395149 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74cc2ac9-6312-46c1-9559-b99ca5c95a60-kube-api-access-s2mkw" (OuterVolumeSpecName: "kube-api-access-s2mkw") pod "74cc2ac9-6312-46c1-9559-b99ca5c95a60" (UID: "74cc2ac9-6312-46c1-9559-b99ca5c95a60"). InnerVolumeSpecName "kube-api-access-s2mkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:22:47.395242 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.395142 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-token" (OuterVolumeSpecName: "istio-token") pod "74cc2ac9-6312-46c1-9559-b99ca5c95a60" (UID: "74cc2ac9-6312-46c1-9559-b99ca5c95a60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:22:47.395242 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.395188 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "74cc2ac9-6312-46c1-9559-b99ca5c95a60" (UID: "74cc2ac9-6312-46c1-9559-b99ca5c95a60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:22:47.494413 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.494361 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-envoy\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:22:47.494413 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.494405 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-workload-certs\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:22:47.494413 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.494414 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2mkw\" (UniqueName: \"kubernetes.io/projected/74cc2ac9-6312-46c1-9559-b99ca5c95a60-kube-api-access-s2mkw\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:22:47.494413 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.494425 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-podinfo\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:22:47.494413 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.494434 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-data\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:22:47.494728 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:47.494442 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74cc2ac9-6312-46c1-9559-b99ca5c95a60-istio-token\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:22:48.008333 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:48.008297 2576 generic.go:356] "Generic (PLEG): container finished" podID="74cc2ac9-6312-46c1-9559-b99ca5c95a60" containerID="5c57d5b595f1d8488eb6e3d51d17fd1e2c5111f0f42b5b6815864d45ffbf710d" exitCode=0 Jul 10 14:22:48.008512 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:48.008340 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" Jul 10 14:22:48.008512 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:48.008383 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" event={"ID":"74cc2ac9-6312-46c1-9559-b99ca5c95a60","Type":"ContainerDied","Data":"5c57d5b595f1d8488eb6e3d51d17fd1e2c5111f0f42b5b6815864d45ffbf710d"} Jul 10 14:22:48.008512 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:48.008439 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8" event={"ID":"74cc2ac9-6312-46c1-9559-b99ca5c95a60","Type":"ContainerDied","Data":"2f0d1c06c7f094e681ba9da6c03aa2fddcb911e45ad0a9c799763b0c8045a65c"} Jul 10 14:22:48.008512 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:48.008464 2576 scope.go:117] "RemoveContainer" containerID="5c57d5b595f1d8488eb6e3d51d17fd1e2c5111f0f42b5b6815864d45ffbf710d" Jul 10 14:22:48.020195 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:48.020085 2576 scope.go:117] "RemoveContainer" containerID="6d56adf284f34794250e40ed4de979205629049e4da565cd078c7aaa30d0593d" Jul 10 14:22:48.029666 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:48.029644 2576 scope.go:117] "RemoveContainer" containerID="5c57d5b595f1d8488eb6e3d51d17fd1e2c5111f0f42b5b6815864d45ffbf710d" Jul 10 14:22:48.030038 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:22:48.030015 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c57d5b595f1d8488eb6e3d51d17fd1e2c5111f0f42b5b6815864d45ffbf710d\": container with ID starting with 5c57d5b595f1d8488eb6e3d51d17fd1e2c5111f0f42b5b6815864d45ffbf710d not found: ID does not exist" containerID="5c57d5b595f1d8488eb6e3d51d17fd1e2c5111f0f42b5b6815864d45ffbf710d" Jul 10 14:22:48.030129 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:48.030048 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c57d5b595f1d8488eb6e3d51d17fd1e2c5111f0f42b5b6815864d45ffbf710d"} err="failed to get container status \"5c57d5b595f1d8488eb6e3d51d17fd1e2c5111f0f42b5b6815864d45ffbf710d\": rpc error: code = NotFound desc = could not find container \"5c57d5b595f1d8488eb6e3d51d17fd1e2c5111f0f42b5b6815864d45ffbf710d\": container with ID starting with 5c57d5b595f1d8488eb6e3d51d17fd1e2c5111f0f42b5b6815864d45ffbf710d not found: ID does not exist" Jul 10 14:22:48.030129 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:48.030068 2576 scope.go:117] "RemoveContainer" containerID="6d56adf284f34794250e40ed4de979205629049e4da565cd078c7aaa30d0593d" Jul 10 14:22:48.030368 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:22:48.030351 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d56adf284f34794250e40ed4de979205629049e4da565cd078c7aaa30d0593d\": container with ID starting with 6d56adf284f34794250e40ed4de979205629049e4da565cd078c7aaa30d0593d not found: ID does not exist" containerID="6d56adf284f34794250e40ed4de979205629049e4da565cd078c7aaa30d0593d" Jul 10 14:22:48.030425 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:48.030373 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d56adf284f34794250e40ed4de979205629049e4da565cd078c7aaa30d0593d"} err="failed to get container status \"6d56adf284f34794250e40ed4de979205629049e4da565cd078c7aaa30d0593d\": rpc error: code = NotFound desc = could not find container \"6d56adf284f34794250e40ed4de979205629049e4da565cd078c7aaa30d0593d\": container with ID starting with 6d56adf284f34794250e40ed4de979205629049e4da565cd078c7aaa30d0593d not found: ID does not exist" Jul 10 14:22:48.031371 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:48.031342 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8"] Jul 10 14:22:48.034934 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:48.034905 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a6e9e1cb-openshift-default-5c47864bf5-zn4x8"] Jul 10 14:22:49.036409 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:49.036372 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74cc2ac9-6312-46c1-9559-b99ca5c95a60" path="/var/lib/kubelet/pods/74cc2ac9-6312-46c1-9559-b99ca5c95a60/volumes" Jul 10 14:22:49.117001 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:49.116961 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:49.237659 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:49.237610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:49.345040 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:49.344944 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:49.453311 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:49.453276 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:53.177884 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:53.177855 2576 scope.go:117] "RemoveContainer" containerID="2f54dc40fa50ac2272680972202955a06c66846a2be366941052d4d963e4f622" Jul 10 14:22:56.168542 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:56.168508 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:22:56.260129 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:22:56.260100 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:23:03.079268 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.079230 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:23:03.373982 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.373893 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4"] Jul 10 14:23:03.374218 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.374206 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a484a423-ef61-4798-bf44-a0cb98767f14" containerName="revoke-keys" Jul 10 14:23:03.374281 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.374219 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a484a423-ef61-4798-bf44-a0cb98767f14" containerName="revoke-keys" Jul 10 14:23:03.374281 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.374229 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74cc2ac9-6312-46c1-9559-b99ca5c95a60" containerName="istio-proxy" Jul 10 14:23:03.374281 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.374234 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="74cc2ac9-6312-46c1-9559-b99ca5c95a60" containerName="istio-proxy" Jul 10 14:23:03.374432 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.374299 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="74cc2ac9-6312-46c1-9559-b99ca5c95a60" containerName="istio-proxy" Jul 10 14:23:03.374432 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.374309 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="74cc2ac9-6312-46c1-9559-b99ca5c95a60" containerName="istio-proxy" Jul 10 14:23:03.374432 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.374316 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a484a423-ef61-4798-bf44-a0cb98767f14" containerName="revoke-keys" Jul 10 14:23:03.377121 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.377099 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.380483 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.380460 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0c74fa54-openshift-default-dockercfg-n5q2d\"" Jul 10 14:23:03.397011 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.396981 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4"] Jul 10 14:23:03.432817 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.432768 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-workload-certs\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.433012 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.432837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-envoy\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.433012 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.432866 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-token\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.433012 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.432933 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-workload-socket\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.433012 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.432962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-data\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.433012 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.432985 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-podinfo\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.433338 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.433017 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-credential-socket\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.433338 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.433057 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1104300-48da-4180-9117-a1a8fcc15ce7-istiod-ca-cert\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.433338 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.433115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shwjn\" (UniqueName: \"kubernetes.io/projected/f1104300-48da-4180-9117-a1a8fcc15ce7-kube-api-access-shwjn\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.439215 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.439173 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:23:03.533850 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.533795 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1104300-48da-4180-9117-a1a8fcc15ce7-istiod-ca-cert\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.534043 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.533864 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shwjn\" (UniqueName: \"kubernetes.io/projected/f1104300-48da-4180-9117-a1a8fcc15ce7-kube-api-access-shwjn\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.534043 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.533909 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-workload-certs\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.534043 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.533926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-envoy\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.534043 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.533942 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-token\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.534043 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.533977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-workload-socket\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.534329 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.534166 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-data\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.534329 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.534200 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-podinfo\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.534329 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.534238 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-credential-socket\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.534514 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.534397 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-workload-socket\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.534574 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.534515 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-data\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.534574 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.534557 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-credential-socket\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.534656 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.534628 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-workload-certs\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.534695 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.534670 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1104300-48da-4180-9117-a1a8fcc15ce7-istiod-ca-cert\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.536335 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.536304 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-envoy\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.536903 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.536881 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-podinfo\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.546706 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.546678 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-token\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.547226 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.547208 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shwjn\" (UniqueName: \"kubernetes.io/projected/f1104300-48da-4180-9117-a1a8fcc15ce7-kube-api-access-shwjn\") pod \"e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.638333 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.638222 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:23:03.688131 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.688088 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:03.861696 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.861656 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4"] Jul 10 14:23:03.868030 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:23:03.867996 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1104300_48da_4180_9117_a1a8fcc15ce7.slice/crio-8cae48d7c05f7f7d451bc34ea564716dde3d0009ae217c1fd2e8dca714cb064a WatchSource:0}: Error finding container 8cae48d7c05f7f7d451bc34ea564716dde3d0009ae217c1fd2e8dca714cb064a: Status 404 returned error can't find the container with id 8cae48d7c05f7f7d451bc34ea564716dde3d0009ae217c1fd2e8dca714cb064a Jul 10 14:23:03.870077 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.870043 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:23:03.870161 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.870113 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:23:03.870161 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:03.870139 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:23:04.088873 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:04.088819 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" event={"ID":"f1104300-48da-4180-9117-a1a8fcc15ce7","Type":"ContainerStarted","Data":"534af6e2c2692d02ca02d17becbb9792ba97c6b5e2bd4525d57c74f869dfae54"} Jul 10 14:23:04.089360 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:04.088901 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" event={"ID":"f1104300-48da-4180-9117-a1a8fcc15ce7","Type":"ContainerStarted","Data":"8cae48d7c05f7f7d451bc34ea564716dde3d0009ae217c1fd2e8dca714cb064a"} Jul 10 14:23:04.114685 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:04.114624 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" podStartSLOduration=1.114600384 podStartE2EDuration="1.114600384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:23:04.1128915 +0000 UTC m=+1991.576496970" watchObservedRunningTime="2026-07-10 14:23:04.114600384 +0000 UTC m=+1991.578205831" Jul 10 14:23:04.688865 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:04.688829 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:04.693286 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:04.693261 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:05.094678 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:05.094589 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:05.095612 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:05.095591 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:06.050753 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:06.050717 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:23:06.169385 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:06.169354 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:23:06.275069 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:06.275036 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:23:06.491006 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:06.490975 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:23:06.735009 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:06.734975 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:23:06.832018 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:06.831933 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:23:10.567765 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:10.567729 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:23:11.311370 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:11.311333 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:23:19.009353 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:19.009309 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:23:20.096095 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:20.096055 2576 patch_prober.go:28] interesting pod/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 10 14:23:20.096532 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:20.096130 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" podUID="f1104300-48da-4180-9117-a1a8fcc15ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 10 14:23:20.177827 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:20.177781 2576 generic.go:356] "Generic (PLEG): container finished" podID="f1104300-48da-4180-9117-a1a8fcc15ce7" containerID="534af6e2c2692d02ca02d17becbb9792ba97c6b5e2bd4525d57c74f869dfae54" exitCode=0 Jul 10 14:23:20.178002 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:20.177844 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" event={"ID":"f1104300-48da-4180-9117-a1a8fcc15ce7","Type":"ContainerDied","Data":"534af6e2c2692d02ca02d17becbb9792ba97c6b5e2bd4525d57c74f869dfae54"} Jul 10 14:23:20.178252 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:20.178239 2576 scope.go:117] "RemoveContainer" containerID="534af6e2c2692d02ca02d17becbb9792ba97c6b5e2bd4525d57c74f869dfae54" Jul 10 14:23:20.179296 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:20.179270 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:23:20.179366 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:20.179349 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:23:20.179404 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:20.179393 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:23:21.184914 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:21.184875 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" event={"ID":"f1104300-48da-4180-9117-a1a8fcc15ce7","Type":"ContainerStarted","Data":"fb652268a7b74bd9fd79afd37efe5d81f23e951b525380a341cc8aa4219e8df3"} Jul 10 14:23:21.688635 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:21.688599 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:21.688859 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:21.688840 2576 patch_prober.go:28] interesting pod/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 10 14:23:21.688921 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:21.688884 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" podUID="f1104300-48da-4180-9117-a1a8fcc15ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 10 14:23:22.689372 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:22.689333 2576 patch_prober.go:28] interesting pod/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 10 14:23:22.689894 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:22.689404 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" podUID="f1104300-48da-4180-9117-a1a8fcc15ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 10 14:23:23.212358 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:23.212323 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:23:23.688530 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:23.688490 2576 patch_prober.go:28] interesting pod/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 10 14:23:23.688725 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:23.688556 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" podUID="f1104300-48da-4180-9117-a1a8fcc15ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 10 14:23:24.689200 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:24.689161 2576 patch_prober.go:28] interesting pod/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 10 14:23:24.689611 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:24.689225 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" podUID="f1104300-48da-4180-9117-a1a8fcc15ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 10 14:23:25.692955 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:25.692925 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:25.693411 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:25.693191 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:25.693977 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:25.693962 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:23:35.607350 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:35.607310 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:23:38.558645 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:38.558606 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c74fa54-j87j4"] Jul 10 14:23:38.559082 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:38.558990 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74cc2ac9-6312-46c1-9559-b99ca5c95a60" containerName="istio-proxy" Jul 10 14:23:38.559082 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:38.559002 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="74cc2ac9-6312-46c1-9559-b99ca5c95a60" containerName="istio-proxy" Jul 10 14:23:38.561264 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:38.561246 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c74fa54-j87j4" Jul 10 14:23:38.565452 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:38.565428 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 14:23:38.565740 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:38.565726 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8qd8l\"" Jul 10 14:23:38.574274 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:38.574247 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c74fa54-j87j4"] Jul 10 14:23:38.652589 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:38.652554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c74fa54-j87j4\" (UID: \"9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c74fa54-j87j4" Jul 10 14:23:38.753330 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:38.753293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c74fa54-j87j4\" (UID: \"9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c74fa54-j87j4" Jul 10 14:23:38.753956 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:38.753937 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c74fa54-j87j4\" (UID: \"9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c74fa54-j87j4" Jul 10 14:23:38.871615 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:38.871525 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c74fa54-j87j4" Jul 10 14:23:39.009931 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:39.009898 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c74fa54-j87j4"] Jul 10 14:23:39.011521 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:23:39.011487 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a58b6a9_fff8_4a71_9ac4_9ed6be2bc395.slice/crio-ae17021be852a8ad254f6ed8c3e899b27c246061ac0fb522d691bd24d10650ab WatchSource:0}: Error finding container ae17021be852a8ad254f6ed8c3e899b27c246061ac0fb522d691bd24d10650ab: Status 404 returned error can't find the container with id ae17021be852a8ad254f6ed8c3e899b27c246061ac0fb522d691bd24d10650ab Jul 10 14:23:39.273969 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:39.273928 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c74fa54-j87j4" event={"ID":"9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395","Type":"ContainerStarted","Data":"ae17021be852a8ad254f6ed8c3e899b27c246061ac0fb522d691bd24d10650ab"} Jul 10 14:23:40.280186 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:40.280146 2576 generic.go:356] "Generic (PLEG): container finished" podID="9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395" containerID="6577199dd8ce2a837cc7ebe4d810e4ad6894c5934702720bebb00e90011d2258" exitCode=0 Jul 10 14:23:40.280602 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:40.280194 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c74fa54-j87j4" event={"ID":"9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395","Type":"ContainerDied","Data":"6577199dd8ce2a837cc7ebe4d810e4ad6894c5934702720bebb00e90011d2258"} Jul 10 14:23:41.415832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:41.415785 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c74fa54-j87j4" Jul 10 14:23:41.474778 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:41.474745 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395-maas-api-service-ca\") pod \"9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395\" (UID: \"9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395\") " Jul 10 14:23:41.475120 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:41.475094 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395" (UID: "9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:23:41.575904 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:41.575789 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395-maas-api-service-ca\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:23:42.129046 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:42.129012 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:23:42.228120 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:42.228076 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:23:42.291688 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:42.291660 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c74fa54-j87j4" Jul 10 14:23:42.291884 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:42.291693 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c74fa54-j87j4" event={"ID":"9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395","Type":"ContainerDied","Data":"ae17021be852a8ad254f6ed8c3e899b27c246061ac0fb522d691bd24d10650ab"} Jul 10 14:23:42.291884 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:42.291739 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae17021be852a8ad254f6ed8c3e899b27c246061ac0fb522d691bd24d10650ab" Jul 10 14:23:49.133329 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:49.133283 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:23:52.646644 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:52.646607 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:23:53.200644 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:53.200614 2576 scope.go:117] "RemoveContainer" containerID="c701b4121933c38ca9a7ff30be4777ccd91e16b952c9c867f3e54cccc6c8ab32" Jul 10 14:23:53.688654 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:53.688617 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:23:58.737966 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:23:58.737935 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:24:17.062948 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:17.062860 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:24:17.195035 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:17.195000 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:24:36.220521 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:36.220491 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:24:37.601775 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:37.601737 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:24:42.725293 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:42.725240 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:24:49.415042 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:49.415007 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:24:49.526390 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:49.526353 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:24:53.104119 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:53.104074 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 14:24:53.107921 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:53.107897 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 14:24:55.966566 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:55.966533 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4"] Jul 10 14:24:55.967133 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:55.966803 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" podUID="f1104300-48da-4180-9117-a1a8fcc15ce7" containerName="istio-proxy" containerID="cri-o://fb652268a7b74bd9fd79afd37efe5d81f23e951b525380a341cc8aa4219e8df3" gracePeriod=30 Jul 10 14:24:56.009489 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:56.009452 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:24:56.222204 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:56.222117 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:24:57.233021 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:57.232987 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:24:57.330651 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:57.330616 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:24:57.433693 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:57.433658 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:24:58.359656 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:58.359621 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:24:58.512327 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:58.512288 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:24:59.746950 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:24:59.746915 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:00.513946 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:00.513910 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:25:01.236222 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.236145 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:25:01.368831 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.368772 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-data\") pod \"f1104300-48da-4180-9117-a1a8fcc15ce7\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " Jul 10 14:25:01.369120 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.368853 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-credential-socket\") pod \"f1104300-48da-4180-9117-a1a8fcc15ce7\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " Jul 10 14:25:01.369120 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.368872 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shwjn\" (UniqueName: \"kubernetes.io/projected/f1104300-48da-4180-9117-a1a8fcc15ce7-kube-api-access-shwjn\") pod \"f1104300-48da-4180-9117-a1a8fcc15ce7\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " Jul 10 14:25:01.369120 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.368899 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-envoy\") pod \"f1104300-48da-4180-9117-a1a8fcc15ce7\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " Jul 10 14:25:01.369120 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.368929 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-workload-certs\") pod \"f1104300-48da-4180-9117-a1a8fcc15ce7\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " Jul 10 14:25:01.369120 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.368952 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1104300-48da-4180-9117-a1a8fcc15ce7-istiod-ca-cert\") pod \"f1104300-48da-4180-9117-a1a8fcc15ce7\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " Jul 10 14:25:01.369120 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.368987 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-workload-socket\") pod \"f1104300-48da-4180-9117-a1a8fcc15ce7\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " Jul 10 14:25:01.369120 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.369067 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-token\") pod \"f1104300-48da-4180-9117-a1a8fcc15ce7\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " Jul 10 14:25:01.369120 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.369113 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-podinfo\") pod \"f1104300-48da-4180-9117-a1a8fcc15ce7\" (UID: \"f1104300-48da-4180-9117-a1a8fcc15ce7\") " Jul 10 14:25:01.369534 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.369114 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f1104300-48da-4180-9117-a1a8fcc15ce7" (UID: "f1104300-48da-4180-9117-a1a8fcc15ce7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:25:01.369534 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.369189 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f1104300-48da-4180-9117-a1a8fcc15ce7" (UID: "f1104300-48da-4180-9117-a1a8fcc15ce7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:25:01.369534 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.369261 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f1104300-48da-4180-9117-a1a8fcc15ce7" (UID: "f1104300-48da-4180-9117-a1a8fcc15ce7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:25:01.369534 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.369410 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-workload-certs\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:25:01.369534 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.369422 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1104300-48da-4180-9117-a1a8fcc15ce7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f1104300-48da-4180-9117-a1a8fcc15ce7" (UID: "f1104300-48da-4180-9117-a1a8fcc15ce7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:25:01.369534 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.369428 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-workload-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:25:01.369534 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.369452 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-credential-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:25:01.369920 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.369634 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-data" (OuterVolumeSpecName: "istio-data") pod "f1104300-48da-4180-9117-a1a8fcc15ce7" (UID: "f1104300-48da-4180-9117-a1a8fcc15ce7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:25:01.371360 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.371331 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f1104300-48da-4180-9117-a1a8fcc15ce7" (UID: "f1104300-48da-4180-9117-a1a8fcc15ce7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:25:01.371486 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.371336 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f1104300-48da-4180-9117-a1a8fcc15ce7" (UID: "f1104300-48da-4180-9117-a1a8fcc15ce7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 14:25:01.371575 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.371559 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1104300-48da-4180-9117-a1a8fcc15ce7-kube-api-access-shwjn" (OuterVolumeSpecName: "kube-api-access-shwjn") pod "f1104300-48da-4180-9117-a1a8fcc15ce7" (UID: "f1104300-48da-4180-9117-a1a8fcc15ce7"). InnerVolumeSpecName "kube-api-access-shwjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:25:01.371620 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.371597 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-token" (OuterVolumeSpecName: "istio-token") pod "f1104300-48da-4180-9117-a1a8fcc15ce7" (UID: "f1104300-48da-4180-9117-a1a8fcc15ce7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:25:01.469897 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.469859 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-token\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:25:01.469897 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.469889 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-podinfo\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:25:01.469897 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.469898 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-data\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:25:01.470147 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.469910 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shwjn\" (UniqueName: \"kubernetes.io/projected/f1104300-48da-4180-9117-a1a8fcc15ce7-kube-api-access-shwjn\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:25:01.470147 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.469918 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1104300-48da-4180-9117-a1a8fcc15ce7-istio-envoy\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:25:01.470147 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.469927 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1104300-48da-4180-9117-a1a8fcc15ce7-istiod-ca-cert\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:25:01.682027 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.681992 2576 generic.go:356] "Generic (PLEG): container finished" podID="f1104300-48da-4180-9117-a1a8fcc15ce7" containerID="fb652268a7b74bd9fd79afd37efe5d81f23e951b525380a341cc8aa4219e8df3" exitCode=0 Jul 10 14:25:01.682241 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.682043 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" Jul 10 14:25:01.682241 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.682079 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" event={"ID":"f1104300-48da-4180-9117-a1a8fcc15ce7","Type":"ContainerDied","Data":"fb652268a7b74bd9fd79afd37efe5d81f23e951b525380a341cc8aa4219e8df3"} Jul 10 14:25:01.682241 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.682136 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4" event={"ID":"f1104300-48da-4180-9117-a1a8fcc15ce7","Type":"ContainerDied","Data":"8cae48d7c05f7f7d451bc34ea564716dde3d0009ae217c1fd2e8dca714cb064a"} Jul 10 14:25:01.682241 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.682163 2576 scope.go:117] "RemoveContainer" containerID="fb652268a7b74bd9fd79afd37efe5d81f23e951b525380a341cc8aa4219e8df3" Jul 10 14:25:01.694225 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.693878 2576 scope.go:117] "RemoveContainer" containerID="534af6e2c2692d02ca02d17becbb9792ba97c6b5e2bd4525d57c74f869dfae54" Jul 10 14:25:01.708314 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.708022 2576 scope.go:117] "RemoveContainer" containerID="fb652268a7b74bd9fd79afd37efe5d81f23e951b525380a341cc8aa4219e8df3" Jul 10 14:25:01.709550 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:25:01.708608 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb652268a7b74bd9fd79afd37efe5d81f23e951b525380a341cc8aa4219e8df3\": container with ID starting with fb652268a7b74bd9fd79afd37efe5d81f23e951b525380a341cc8aa4219e8df3 not found: ID does not exist" containerID="fb652268a7b74bd9fd79afd37efe5d81f23e951b525380a341cc8aa4219e8df3" Jul 10 14:25:01.709550 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.708661 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb652268a7b74bd9fd79afd37efe5d81f23e951b525380a341cc8aa4219e8df3"} err="failed to get container status \"fb652268a7b74bd9fd79afd37efe5d81f23e951b525380a341cc8aa4219e8df3\": rpc error: code = NotFound desc = could not find container \"fb652268a7b74bd9fd79afd37efe5d81f23e951b525380a341cc8aa4219e8df3\": container with ID starting with fb652268a7b74bd9fd79afd37efe5d81f23e951b525380a341cc8aa4219e8df3 not found: ID does not exist" Jul 10 14:25:01.709550 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.708686 2576 scope.go:117] "RemoveContainer" containerID="534af6e2c2692d02ca02d17becbb9792ba97c6b5e2bd4525d57c74f869dfae54" Jul 10 14:25:01.710210 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:25:01.709901 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534af6e2c2692d02ca02d17becbb9792ba97c6b5e2bd4525d57c74f869dfae54\": container with ID starting with 534af6e2c2692d02ca02d17becbb9792ba97c6b5e2bd4525d57c74f869dfae54 not found: ID does not exist" containerID="534af6e2c2692d02ca02d17becbb9792ba97c6b5e2bd4525d57c74f869dfae54" Jul 10 14:25:01.710210 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.709934 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534af6e2c2692d02ca02d17becbb9792ba97c6b5e2bd4525d57c74f869dfae54"} err="failed to get container status \"534af6e2c2692d02ca02d17becbb9792ba97c6b5e2bd4525d57c74f869dfae54\": rpc error: code = NotFound desc = could not find container \"534af6e2c2692d02ca02d17becbb9792ba97c6b5e2bd4525d57c74f869dfae54\": container with ID starting with 534af6e2c2692d02ca02d17becbb9792ba97c6b5e2bd4525d57c74f869dfae54 not found: ID does not exist" Jul 10 14:25:01.714185 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.713847 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4"] Jul 10 14:25:01.718257 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:01.718190 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0c74fa54-openshift-default-bfdb9679f-68qt4"] Jul 10 14:25:03.036989 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:03.036955 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1104300-48da-4180-9117-a1a8fcc15ce7" path="/var/lib/kubelet/pods/f1104300-48da-4180-9117-a1a8fcc15ce7/volumes" Jul 10 14:25:06.157631 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:06.157593 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:06.295434 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:06.295402 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:06.857573 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:06.857530 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:13.898470 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:13.898434 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:13.994145 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:13.994093 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:14.098446 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:14.098413 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:14.119235 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:14.119200 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:25:24.660362 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:24.660328 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:24.983097 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:24.983001 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:45.945680 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:45.945594 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:52.755949 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:52.755910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:52.877114 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:52.877082 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:53.099387 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.099292 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:53.226545 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.226517 2576 scope.go:117] "RemoveContainer" containerID="a77ca2100177a9282dbd7000c9b33d4e24dcf118c484fa1e9b59e55fafea7219" Jul 10 14:25:53.403508 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.403475 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2"] Jul 10 14:25:53.403801 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.403789 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1104300-48da-4180-9117-a1a8fcc15ce7" containerName="istio-proxy" Jul 10 14:25:53.403801 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.403803 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1104300-48da-4180-9117-a1a8fcc15ce7" containerName="istio-proxy" Jul 10 14:25:53.403918 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.403826 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395" containerName="revoke-keys" Jul 10 14:25:53.403918 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.403835 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395" containerName="revoke-keys" Jul 10 14:25:53.403918 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.403849 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1104300-48da-4180-9117-a1a8fcc15ce7" containerName="istio-proxy" Jul 10 14:25:53.403918 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.403856 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1104300-48da-4180-9117-a1a8fcc15ce7" containerName="istio-proxy" Jul 10 14:25:53.403918 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.403906 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1104300-48da-4180-9117-a1a8fcc15ce7" containerName="istio-proxy" Jul 10 14:25:53.403918 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.403916 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395" containerName="revoke-keys" Jul 10 14:25:53.408098 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.408074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.410901 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.410878 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ee9f0f-openshift-default-dockercfg-kttkh\"" Jul 10 14:25:53.422524 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.422497 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2"] Jul 10 14:25:53.480068 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.480031 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:53.527407 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.527368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-credential-socket\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.527598 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.527428 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-envoy\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.527598 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.527458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-workload-socket\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.527598 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.527498 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd16bfab-e709-48bc-985c-6b498a3d4020-istiod-ca-cert\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.527598 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.527523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-workload-certs\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.527598 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.527584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-podinfo\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.527802 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.527617 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-token\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.527802 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.527641 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4klcf\" (UniqueName: \"kubernetes.io/projected/fd16bfab-e709-48bc-985c-6b498a3d4020-kube-api-access-4klcf\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.527802 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.527688 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-data\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.629181 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.629141 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-data\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.629379 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.629309 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-credential-socket\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.629379 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.629363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-envoy\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.629505 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.629394 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-workload-socket\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.629505 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.629426 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd16bfab-e709-48bc-985c-6b498a3d4020-istiod-ca-cert\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.629505 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.629444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-workload-certs\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.629505 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.629476 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-podinfo\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.629696 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.629513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-token\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.629696 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.629540 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4klcf\" (UniqueName: \"kubernetes.io/projected/fd16bfab-e709-48bc-985c-6b498a3d4020-kube-api-access-4klcf\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.629696 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.629620 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-data\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.629881 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.629784 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-credential-socket\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.629881 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.629833 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-workload-certs\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.630224 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.630201 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd16bfab-e709-48bc-985c-6b498a3d4020-istiod-ca-cert\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.630313 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.630201 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-workload-socket\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.631892 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.631873 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-envoy\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.632135 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.632116 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-podinfo\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.639060 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.639029 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-token\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.639148 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.639110 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4klcf\" (UniqueName: \"kubernetes.io/projected/fd16bfab-e709-48bc-985c-6b498a3d4020-kube-api-access-4klcf\") pod \"e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.720612 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.720505 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:53.874858 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.874823 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2"] Jul 10 14:25:53.878002 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:25:53.877974 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd16bfab_e709_48bc_985c_6b498a3d4020.slice/crio-d1a7f8a8596d82883dd8704164574c55fffee8f465657aceb3dcd050e5b98550 WatchSource:0}: Error finding container d1a7f8a8596d82883dd8704164574c55fffee8f465657aceb3dcd050e5b98550: Status 404 returned error can't find the container with id d1a7f8a8596d82883dd8704164574c55fffee8f465657aceb3dcd050e5b98550 Jul 10 14:25:53.880557 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.880524 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:25:53.880653 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.880597 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:25:53.880653 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.880626 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:25:53.930746 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:53.930713 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" event={"ID":"fd16bfab-e709-48bc-985c-6b498a3d4020","Type":"ContainerStarted","Data":"d1a7f8a8596d82883dd8704164574c55fffee8f465657aceb3dcd050e5b98550"} Jul 10 14:25:54.119143 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:54.119060 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:54.936643 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:54.936609 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" event={"ID":"fd16bfab-e709-48bc-985c-6b498a3d4020","Type":"ContainerStarted","Data":"9aaafe411bf493b5dc08bf6b57a71dca49cf112be9f11adbb3ee6820f53240f6"} Jul 10 14:25:54.967256 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:54.967205 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" podStartSLOduration=1.967189398 podStartE2EDuration="1.967189398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:25:54.964619909 +0000 UTC m=+2162.428225378" watchObservedRunningTime="2026-07-10 14:25:54.967189398 +0000 UTC m=+2162.430794877" Jul 10 14:25:55.721001 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:55.720966 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:55.725946 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:55.725918 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:55.941400 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:55.941371 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:55.942526 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:55.942501 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:25:56.938049 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:56.938009 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:57.077971 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:57.077938 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:57.261840 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:57.261727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:57.398756 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:57.398724 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:57.785202 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:57.785152 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:25:57.886304 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:25:57.886264 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:01.525428 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:01.525379 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:01.859305 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:01.859204 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:02.192964 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.192920 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q"] Jul 10 14:26:02.193392 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.193374 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1104300-48da-4180-9117-a1a8fcc15ce7" containerName="istio-proxy" Jul 10 14:26:02.196412 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.196387 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.199368 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.199343 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e3d9d1-openshift-default-dockercfg-gf6jv\"" Jul 10 14:26:02.222148 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.222108 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q"] Jul 10 14:26:02.258069 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.258032 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:02.309290 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.309251 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-credential-socket\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.309290 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.309301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-workload-socket\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.309575 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.309329 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-istio-envoy\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.309575 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.309413 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/118c0028-efd7-45bc-907b-11afad71fec3-istiod-ca-cert\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.309575 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.309457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-istio-data\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.309575 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.309502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rkcq\" (UniqueName: \"kubernetes.io/projected/118c0028-efd7-45bc-907b-11afad71fec3-kube-api-access-7rkcq\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.309575 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.309541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/118c0028-efd7-45bc-907b-11afad71fec3-istio-token\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.309832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.309576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-workload-certs\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.309832 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.309609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/118c0028-efd7-45bc-907b-11afad71fec3-istio-podinfo\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.410700 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.410657 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-credential-socket\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.410700 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.410699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-workload-socket\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.410987 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.410717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-istio-envoy\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.410987 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.410850 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/118c0028-efd7-45bc-907b-11afad71fec3-istiod-ca-cert\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.410987 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.410891 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-istio-data\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.410987 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.410933 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rkcq\" (UniqueName: \"kubernetes.io/projected/118c0028-efd7-45bc-907b-11afad71fec3-kube-api-access-7rkcq\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.410987 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.410977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/118c0028-efd7-45bc-907b-11afad71fec3-istio-token\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.411210 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.411012 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-workload-certs\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.411210 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.411046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/118c0028-efd7-45bc-907b-11afad71fec3-istio-podinfo\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.411466 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.411441 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-workload-socket\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.411599 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.411496 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-istio-data\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.411599 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.411531 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-credential-socket\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.411599 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.411567 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-workload-certs\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.411858 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.411801 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/118c0028-efd7-45bc-907b-11afad71fec3-istiod-ca-cert\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.413199 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.413183 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-istio-envoy\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.413665 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.413647 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/118c0028-efd7-45bc-907b-11afad71fec3-istio-podinfo\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.423055 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.423028 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rkcq\" (UniqueName: \"kubernetes.io/projected/118c0028-efd7-45bc-907b-11afad71fec3-kube-api-access-7rkcq\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.423222 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.423194 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/118c0028-efd7-45bc-907b-11afad71fec3-istio-token\") pod \"e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.469431 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.469343 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:02.507466 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.507427 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:02.661184 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.661157 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q"] Jul 10 14:26:02.662670 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:26:02.662640 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118c0028_efd7_45bc_907b_11afad71fec3.slice/crio-548c4b7a9cae11a2f9ca0b274867787f009731d97d75c335f7d6b81e050e7d1d WatchSource:0}: Error finding container 548c4b7a9cae11a2f9ca0b274867787f009731d97d75c335f7d6b81e050e7d1d: Status 404 returned error can't find the container with id 548c4b7a9cae11a2f9ca0b274867787f009731d97d75c335f7d6b81e050e7d1d Jul 10 14:26:02.664764 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.664734 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:26:02.664857 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.664829 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:26:02.664902 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.664857 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:26:02.977723 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.977621 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" event={"ID":"118c0028-efd7-45bc-907b-11afad71fec3","Type":"ContainerStarted","Data":"bdfdbb9b6a074e361e62e6fac1ed52d9782d003e008e4f4b9062c9f4efd384a4"} Jul 10 14:26:02.977723 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:02.977684 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" event={"ID":"118c0028-efd7-45bc-907b-11afad71fec3","Type":"ContainerStarted","Data":"548c4b7a9cae11a2f9ca0b274867787f009731d97d75c335f7d6b81e050e7d1d"} Jul 10 14:26:03.017741 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:03.017676 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" podStartSLOduration=1.017651605 podStartE2EDuration="1.017651605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:26:03.016600036 +0000 UTC m=+2170.480205495" watchObservedRunningTime="2026-07-10 14:26:03.017651605 +0000 UTC m=+2170.481257052" Jul 10 14:26:03.507768 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:03.507724 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:03.512837 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:03.512785 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:03.983171 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:03.983133 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:03.984267 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:03.984246 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:04.994781 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:04.994751 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:26:05.140044 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:05.140013 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:05.247729 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:05.247638 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:05.532679 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:05.532595 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:05.829850 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:05.829737 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:05.952561 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:05.952525 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:09.531613 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:09.531565 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:11.983021 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:11.982973 2576 patch_prober.go:28] interesting pod/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 14:26:11.983420 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:11.983050 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" podUID="fd16bfab-e709-48bc-985c-6b498a3d4020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 14:26:13.052727 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:13.052683 2576 generic.go:356] "Generic (PLEG): container finished" podID="fd16bfab-e709-48bc-985c-6b498a3d4020" containerID="9aaafe411bf493b5dc08bf6b57a71dca49cf112be9f11adbb3ee6820f53240f6" exitCode=0 Jul 10 14:26:13.054756 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:13.054722 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" event={"ID":"fd16bfab-e709-48bc-985c-6b498a3d4020","Type":"ContainerDied","Data":"9aaafe411bf493b5dc08bf6b57a71dca49cf112be9f11adbb3ee6820f53240f6"} Jul 10 14:26:13.055163 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:13.055148 2576 scope.go:117] "RemoveContainer" containerID="9aaafe411bf493b5dc08bf6b57a71dca49cf112be9f11adbb3ee6820f53240f6" Jul 10 14:26:13.056346 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:13.056315 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:26:13.056414 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:13.056386 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:26:13.056451 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:13.056418 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:26:14.063830 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:14.063766 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" event={"ID":"fd16bfab-e709-48bc-985c-6b498a3d4020","Type":"ContainerStarted","Data":"f7e48e070d60d189016a75bdcc72031b0880272384dc59d5b2da85f8be23faf1"} Jul 10 14:26:14.720963 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:14.720915 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:26:14.721162 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:14.721090 2576 patch_prober.go:28] interesting pod/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 10 14:26:14.721162 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:14.721132 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" podUID="fd16bfab-e709-48bc-985c-6b498a3d4020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 10 14:26:15.721219 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:15.721176 2576 patch_prober.go:28] interesting pod/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 10 14:26:15.721712 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:15.721245 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" podUID="fd16bfab-e709-48bc-985c-6b498a3d4020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 10 14:26:16.202726 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:16.202679 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:26:16.721615 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:16.721580 2576 patch_prober.go:28] interesting pod/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 10 14:26:16.722099 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:16.721640 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" podUID="fd16bfab-e709-48bc-985c-6b498a3d4020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 10 14:26:17.721279 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:17.721234 2576 patch_prober.go:28] interesting pod/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 10 14:26:17.721488 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:17.721306 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" podUID="fd16bfab-e709-48bc-985c-6b498a3d4020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 10 14:26:18.690451 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:18.690418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:18.725370 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:18.725333 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:26:18.725626 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:18.725600 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:26:18.726271 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:18.726256 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:26:18.785239 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:18.785188 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:19.694192 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:19.693170 2576 patch_prober.go:28] interesting pod/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": read tcp 10.133.0.2:52874->10.133.0.47:15021: read: connection reset by peer" start-of-body= Jul 10 14:26:19.694192 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:19.693248 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" podUID="118c0028-efd7-45bc-907b-11afad71fec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": read tcp 10.133.0.2:52874->10.133.0.47:15021: read: connection reset by peer" Jul 10 14:26:20.105346 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:20.105247 2576 generic.go:356] "Generic (PLEG): container finished" podID="118c0028-efd7-45bc-907b-11afad71fec3" containerID="bdfdbb9b6a074e361e62e6fac1ed52d9782d003e008e4f4b9062c9f4efd384a4" exitCode=0 Jul 10 14:26:20.105346 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:20.105323 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" event={"ID":"118c0028-efd7-45bc-907b-11afad71fec3","Type":"ContainerDied","Data":"bdfdbb9b6a074e361e62e6fac1ed52d9782d003e008e4f4b9062c9f4efd384a4"} Jul 10 14:26:20.105879 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:20.105850 2576 scope.go:117] "RemoveContainer" containerID="bdfdbb9b6a074e361e62e6fac1ed52d9782d003e008e4f4b9062c9f4efd384a4" Jul 10 14:26:20.106861 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:20.106836 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:26:20.106954 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:20.106901 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:26:20.106954 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:20.106936 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:26:21.112499 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:21.112465 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" event={"ID":"118c0028-efd7-45bc-907b-11afad71fec3","Type":"ContainerStarted","Data":"fd43afae6b810fb87762f0b0992022befeb9a2c8742f71945cd487e0440851c5"} Jul 10 14:26:21.508554 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:21.508511 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:21.508888 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:21.508864 2576 patch_prober.go:28] interesting pod/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 10 14:26:21.508966 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:21.508914 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" podUID="118c0028-efd7-45bc-907b-11afad71fec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 10 14:26:22.508891 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:22.508843 2576 patch_prober.go:28] interesting pod/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 10 14:26:22.509290 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:22.508908 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" podUID="118c0028-efd7-45bc-907b-11afad71fec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 10 14:26:23.508606 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:23.508563 2576 patch_prober.go:28] interesting pod/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 10 14:26:23.508835 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:23.508630 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" podUID="118c0028-efd7-45bc-907b-11afad71fec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 10 14:26:24.508016 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:24.507974 2576 patch_prober.go:28] interesting pod/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 10 14:26:24.508494 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:24.508040 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" podUID="118c0028-efd7-45bc-907b-11afad71fec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 10 14:26:25.535928 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:25.535866 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:25.536406 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:25.536224 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:25.536940 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:25.536917 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:26:27.104192 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:27.104158 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:27.237235 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:27.237190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:31.370225 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:31.370186 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:35.325142 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:35.325099 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:26:46.142722 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:46.142678 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:26:51.805510 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:51.805473 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:51.924540 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:51.924509 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:53.030478 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:53.030440 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:26:56.766995 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:26:56.766955 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:27:02.195644 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:02.195604 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:27:02.310709 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:02.310676 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:27:10.357838 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:10.357791 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:27:13.937894 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:13.937863 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:27:15.147533 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:15.147499 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:27:15.301376 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:15.301342 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm"] Jul 10 14:27:15.304929 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:15.304911 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm" Jul 10 14:27:15.320988 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:15.320958 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 14:27:15.321424 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:15.321407 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8qd8l\"" Jul 10 14:27:15.398979 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:15.398884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00839295-b0d3-414b-a161-9d210f3e8917-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm\" (UID: \"00839295-b0d3-414b-a161-9d210f3e8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm" Jul 10 14:27:15.402143 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:15.402113 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm"] Jul 10 14:27:15.500112 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:15.500075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00839295-b0d3-414b-a161-9d210f3e8917-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm\" (UID: \"00839295-b0d3-414b-a161-9d210f3e8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm" Jul 10 14:27:15.500751 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:15.500727 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00839295-b0d3-414b-a161-9d210f3e8917-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm\" (UID: \"00839295-b0d3-414b-a161-9d210f3e8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm" Jul 10 14:27:15.615361 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:15.615321 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm" Jul 10 14:27:15.774040 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:15.774008 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm"] Jul 10 14:27:15.776635 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:27:15.776608 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00839295_b0d3_414b_a161_9d210f3e8917.slice/crio-53e240842fd7e875ac48840e7fe7a19eaae0b1c4c87672ef54dfe31d28193c09 WatchSource:0}: Error finding container 53e240842fd7e875ac48840e7fe7a19eaae0b1c4c87672ef54dfe31d28193c09: Status 404 returned error can't find the container with id 53e240842fd7e875ac48840e7fe7a19eaae0b1c4c87672ef54dfe31d28193c09 Jul 10 14:27:16.391688 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:16.391650 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm" event={"ID":"00839295-b0d3-414b-a161-9d210f3e8917","Type":"ContainerStarted","Data":"3d85bf824430dc9c54d30aa7b376a0d340c109db862c53e924718d86f4bc377b"} Jul 10 14:27:16.392105 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:16.391711 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm" event={"ID":"00839295-b0d3-414b-a161-9d210f3e8917","Type":"ContainerStarted","Data":"53e240842fd7e875ac48840e7fe7a19eaae0b1c4c87672ef54dfe31d28193c09"} Jul 10 14:27:16.438575 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:16.438522 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm" podStartSLOduration=1.438502644 podStartE2EDuration="1.438502644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:27:16.434823115 +0000 UTC m=+2243.898428559" watchObservedRunningTime="2026-07-10 14:27:16.438502644 +0000 UTC m=+2243.902108089" Jul 10 14:27:16.444710 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:16.444595 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:27:16.551927 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:16.551898 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:27:17.398732 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:17.398699 2576 generic.go:356] "Generic (PLEG): container finished" podID="00839295-b0d3-414b-a161-9d210f3e8917" containerID="3d85bf824430dc9c54d30aa7b376a0d340c109db862c53e924718d86f4bc377b" exitCode=0 Jul 10 14:27:17.399155 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:17.398784 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm" event={"ID":"00839295-b0d3-414b-a161-9d210f3e8917","Type":"ContainerDied","Data":"3d85bf824430dc9c54d30aa7b376a0d340c109db862c53e924718d86f4bc377b"} Jul 10 14:27:18.533855 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:18.533826 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm" Jul 10 14:27:18.629465 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:18.629429 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00839295-b0d3-414b-a161-9d210f3e8917-maas-api-service-ca\") pod \"00839295-b0d3-414b-a161-9d210f3e8917\" (UID: \"00839295-b0d3-414b-a161-9d210f3e8917\") " Jul 10 14:27:18.629802 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:18.629779 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00839295-b0d3-414b-a161-9d210f3e8917-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00839295-b0d3-414b-a161-9d210f3e8917" (UID: "00839295-b0d3-414b-a161-9d210f3e8917"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:27:18.730263 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:18.730182 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00839295-b0d3-414b-a161-9d210f3e8917-maas-api-service-ca\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:27:19.410328 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:19.410296 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm" Jul 10 14:27:19.410328 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:19.410310 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm" event={"ID":"00839295-b0d3-414b-a161-9d210f3e8917","Type":"ContainerDied","Data":"53e240842fd7e875ac48840e7fe7a19eaae0b1c4c87672ef54dfe31d28193c09"} Jul 10 14:27:19.410594 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:19.410354 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53e240842fd7e875ac48840e7fe7a19eaae0b1c4c87672ef54dfe31d28193c09" Jul 10 14:27:26.847963 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:26.847918 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:27:29.682084 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:29.682044 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:27:30.522332 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:30.522296 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:27:39.513006 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:39.512957 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:27:39.661006 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:39.660972 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:27:41.546096 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:41.546055 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2"] Jul 10 14:27:41.546559 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:41.546271 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" podUID="fd16bfab-e709-48bc-985c-6b498a3d4020" containerName="istio-proxy" containerID="cri-o://f7e48e070d60d189016a75bdcc72031b0880272384dc59d5b2da85f8be23faf1" gracePeriod=30 Jul 10 14:27:41.623611 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:41.623566 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:27:41.848860 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:41.848722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:27:43.013396 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:43.013365 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:27:43.148513 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:43.148475 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:27:46.804321 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.804293 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:27:46.890907 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.890801 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-podinfo\") pod \"fd16bfab-e709-48bc-985c-6b498a3d4020\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " Jul 10 14:27:46.890907 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.890877 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-envoy\") pod \"fd16bfab-e709-48bc-985c-6b498a3d4020\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " Jul 10 14:27:46.891155 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.890909 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-credential-socket\") pod \"fd16bfab-e709-48bc-985c-6b498a3d4020\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " Jul 10 14:27:46.891155 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.890931 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd16bfab-e709-48bc-985c-6b498a3d4020-istiod-ca-cert\") pod \"fd16bfab-e709-48bc-985c-6b498a3d4020\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " Jul 10 14:27:46.891155 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.890958 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-data\") pod \"fd16bfab-e709-48bc-985c-6b498a3d4020\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " Jul 10 14:27:46.891155 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.891007 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-workload-socket\") pod \"fd16bfab-e709-48bc-985c-6b498a3d4020\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " Jul 10 14:27:46.891155 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.891034 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-workload-certs\") pod \"fd16bfab-e709-48bc-985c-6b498a3d4020\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " Jul 10 14:27:46.891155 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.891052 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-token\") pod \"fd16bfab-e709-48bc-985c-6b498a3d4020\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " Jul 10 14:27:46.891155 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.891077 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4klcf\" (UniqueName: \"kubernetes.io/projected/fd16bfab-e709-48bc-985c-6b498a3d4020-kube-api-access-4klcf\") pod \"fd16bfab-e709-48bc-985c-6b498a3d4020\" (UID: \"fd16bfab-e709-48bc-985c-6b498a3d4020\") " Jul 10 14:27:46.891509 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.891201 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fd16bfab-e709-48bc-985c-6b498a3d4020" (UID: "fd16bfab-e709-48bc-985c-6b498a3d4020"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:27:46.891509 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.891397 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-credential-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:27:46.891509 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.891435 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fd16bfab-e709-48bc-985c-6b498a3d4020" (UID: "fd16bfab-e709-48bc-985c-6b498a3d4020"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:27:46.891509 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.891442 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd16bfab-e709-48bc-985c-6b498a3d4020-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fd16bfab-e709-48bc-985c-6b498a3d4020" (UID: "fd16bfab-e709-48bc-985c-6b498a3d4020"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:27:46.891727 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.891523 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fd16bfab-e709-48bc-985c-6b498a3d4020" (UID: "fd16bfab-e709-48bc-985c-6b498a3d4020"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:27:46.891880 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.891857 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-data" (OuterVolumeSpecName: "istio-data") pod "fd16bfab-e709-48bc-985c-6b498a3d4020" (UID: "fd16bfab-e709-48bc-985c-6b498a3d4020"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:27:46.893220 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.893193 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fd16bfab-e709-48bc-985c-6b498a3d4020" (UID: "fd16bfab-e709-48bc-985c-6b498a3d4020"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:27:46.893322 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.893232 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd16bfab-e709-48bc-985c-6b498a3d4020-kube-api-access-4klcf" (OuterVolumeSpecName: "kube-api-access-4klcf") pod "fd16bfab-e709-48bc-985c-6b498a3d4020" (UID: "fd16bfab-e709-48bc-985c-6b498a3d4020"). InnerVolumeSpecName "kube-api-access-4klcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:27:46.893459 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.893438 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fd16bfab-e709-48bc-985c-6b498a3d4020" (UID: "fd16bfab-e709-48bc-985c-6b498a3d4020"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 14:27:46.893611 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.893590 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-token" (OuterVolumeSpecName: "istio-token") pod "fd16bfab-e709-48bc-985c-6b498a3d4020" (UID: "fd16bfab-e709-48bc-985c-6b498a3d4020"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:27:46.992421 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.992381 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-workload-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:27:46.992421 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.992413 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-workload-certs\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:27:46.992421 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.992424 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-token\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:27:46.992421 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.992432 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4klcf\" (UniqueName: \"kubernetes.io/projected/fd16bfab-e709-48bc-985c-6b498a3d4020-kube-api-access-4klcf\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:27:46.992700 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.992441 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-podinfo\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:27:46.992700 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.992451 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-envoy\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:27:46.992700 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.992458 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd16bfab-e709-48bc-985c-6b498a3d4020-istiod-ca-cert\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:27:46.992700 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:46.992465 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd16bfab-e709-48bc-985c-6b498a3d4020-istio-data\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:27:47.548261 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:47.548225 2576 generic.go:356] "Generic (PLEG): container finished" podID="fd16bfab-e709-48bc-985c-6b498a3d4020" containerID="f7e48e070d60d189016a75bdcc72031b0880272384dc59d5b2da85f8be23faf1" exitCode=0 Jul 10 14:27:47.548449 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:47.548276 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" Jul 10 14:27:47.548449 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:47.548309 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" event={"ID":"fd16bfab-e709-48bc-985c-6b498a3d4020","Type":"ContainerDied","Data":"f7e48e070d60d189016a75bdcc72031b0880272384dc59d5b2da85f8be23faf1"} Jul 10 14:27:47.548449 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:47.548365 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2" event={"ID":"fd16bfab-e709-48bc-985c-6b498a3d4020","Type":"ContainerDied","Data":"d1a7f8a8596d82883dd8704164574c55fffee8f465657aceb3dcd050e5b98550"} Jul 10 14:27:47.548449 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:47.548391 2576 scope.go:117] "RemoveContainer" containerID="f7e48e070d60d189016a75bdcc72031b0880272384dc59d5b2da85f8be23faf1" Jul 10 14:27:47.557106 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:47.557091 2576 scope.go:117] "RemoveContainer" containerID="9aaafe411bf493b5dc08bf6b57a71dca49cf112be9f11adbb3ee6820f53240f6" Jul 10 14:27:47.565597 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:47.565576 2576 scope.go:117] "RemoveContainer" containerID="f7e48e070d60d189016a75bdcc72031b0880272384dc59d5b2da85f8be23faf1" Jul 10 14:27:47.565939 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:27:47.565916 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e48e070d60d189016a75bdcc72031b0880272384dc59d5b2da85f8be23faf1\": container with ID starting with f7e48e070d60d189016a75bdcc72031b0880272384dc59d5b2da85f8be23faf1 not found: ID does not exist" containerID="f7e48e070d60d189016a75bdcc72031b0880272384dc59d5b2da85f8be23faf1" Jul 10 14:27:47.566039 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:47.565948 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e48e070d60d189016a75bdcc72031b0880272384dc59d5b2da85f8be23faf1"} err="failed to get container status \"f7e48e070d60d189016a75bdcc72031b0880272384dc59d5b2da85f8be23faf1\": rpc error: code = NotFound desc = could not find container \"f7e48e070d60d189016a75bdcc72031b0880272384dc59d5b2da85f8be23faf1\": container with ID starting with f7e48e070d60d189016a75bdcc72031b0880272384dc59d5b2da85f8be23faf1 not found: ID does not exist" Jul 10 14:27:47.566039 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:47.565966 2576 scope.go:117] "RemoveContainer" containerID="9aaafe411bf493b5dc08bf6b57a71dca49cf112be9f11adbb3ee6820f53240f6" Jul 10 14:27:47.566237 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:27:47.566218 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aaafe411bf493b5dc08bf6b57a71dca49cf112be9f11adbb3ee6820f53240f6\": container with ID starting with 9aaafe411bf493b5dc08bf6b57a71dca49cf112be9f11adbb3ee6820f53240f6 not found: ID does not exist" containerID="9aaafe411bf493b5dc08bf6b57a71dca49cf112be9f11adbb3ee6820f53240f6" Jul 10 14:27:47.566278 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:47.566244 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aaafe411bf493b5dc08bf6b57a71dca49cf112be9f11adbb3ee6820f53240f6"} err="failed to get container status \"9aaafe411bf493b5dc08bf6b57a71dca49cf112be9f11adbb3ee6820f53240f6\": rpc error: code = NotFound desc = could not find container \"9aaafe411bf493b5dc08bf6b57a71dca49cf112be9f11adbb3ee6820f53240f6\": container with ID starting with 9aaafe411bf493b5dc08bf6b57a71dca49cf112be9f11adbb3ee6820f53240f6 not found: ID does not exist" Jul 10 14:27:47.577706 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:47.577674 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2"] Jul 10 14:27:47.589701 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:47.589672 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ee9f0f-openshift-default-9cc976b8c-8wxv2"] Jul 10 14:27:49.037226 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:49.037190 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd16bfab-e709-48bc-985c-6b498a3d4020" path="/var/lib/kubelet/pods/fd16bfab-e709-48bc-985c-6b498a3d4020/volumes" Jul 10 14:27:52.989954 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:52.989913 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:27:56.349033 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:56.348998 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:27:57.117266 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:27:57.117236 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:05.469667 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:05.469630 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:05.685696 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:05.685650 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:08.050133 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:08.050098 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q"] Jul 10 14:28:08.050625 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:08.050301 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" podUID="118c0028-efd7-45bc-907b-11afad71fec3" containerName="istio-proxy" containerID="cri-o://fd43afae6b810fb87762f0b0992022befeb9a2c8742f71945cd487e0440851c5" gracePeriod=30 Jul 10 14:28:08.087826 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:08.087786 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:08.219475 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:08.219433 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:08.354910 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:08.354795 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:08.768262 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:08.768222 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:09.246018 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.245978 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx"] Jul 10 14:28:09.246437 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.246351 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd16bfab-e709-48bc-985c-6b498a3d4020" containerName="istio-proxy" Jul 10 14:28:09.246437 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.246365 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd16bfab-e709-48bc-985c-6b498a3d4020" containerName="istio-proxy" Jul 10 14:28:09.246437 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.246390 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00839295-b0d3-414b-a161-9d210f3e8917" containerName="revoke-keys" Jul 10 14:28:09.246437 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.246396 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="00839295-b0d3-414b-a161-9d210f3e8917" containerName="revoke-keys" Jul 10 14:28:09.246568 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.246449 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd16bfab-e709-48bc-985c-6b498a3d4020" containerName="istio-proxy" Jul 10 14:28:09.246568 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.246457 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="00839295-b0d3-414b-a161-9d210f3e8917" containerName="revoke-keys" Jul 10 14:28:09.246568 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.246465 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd16bfab-e709-48bc-985c-6b498a3d4020" containerName="istio-proxy" Jul 10 14:28:09.249405 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.249386 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.253714 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.253690 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7beb38-openshift-default-dockercfg-pr2mb\"" Jul 10 14:28:09.267491 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.267458 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx"] Jul 10 14:28:09.276085 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.276051 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:09.391597 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.391561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/977d57fe-4c6f-4520-882a-386adc295618-istiod-ca-cert\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.391785 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.391599 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/977d57fe-4c6f-4520-882a-386adc295618-istio-podinfo\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.392122 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.392095 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8f6l\" (UniqueName: \"kubernetes.io/projected/977d57fe-4c6f-4520-882a-386adc295618-kube-api-access-c8f6l\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.392339 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.392322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-istio-data\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.392462 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.392447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-credential-socket\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.392625 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.392611 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-workload-certs\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.392731 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.392718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-istio-envoy\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.392859 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.392844 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/977d57fe-4c6f-4520-882a-386adc295618-istio-token\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.392989 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.392976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-workload-socket\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.478077 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.478038 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:09.494205 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.494168 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/977d57fe-4c6f-4520-882a-386adc295618-istiod-ca-cert\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.494205 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.494209 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/977d57fe-4c6f-4520-882a-386adc295618-istio-podinfo\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.494470 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.494229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8f6l\" (UniqueName: \"kubernetes.io/projected/977d57fe-4c6f-4520-882a-386adc295618-kube-api-access-c8f6l\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.494470 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.494252 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-istio-data\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.494470 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.494275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-credential-socket\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.494470 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.494400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-workload-certs\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.494470 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.494442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-istio-envoy\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.494470 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.494466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/977d57fe-4c6f-4520-882a-386adc295618-istio-token\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.494770 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.494487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-workload-socket\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.494770 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.494695 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-istio-data\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.494770 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.494741 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-workload-certs\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.494935 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.494890 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-workload-socket\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.495009 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.494990 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/977d57fe-4c6f-4520-882a-386adc295618-istiod-ca-cert\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.495046 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.494993 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-credential-socket\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.496845 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.496754 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/977d57fe-4c6f-4520-882a-386adc295618-istio-podinfo\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.496953 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.496840 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-istio-envoy\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.515430 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.515395 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8f6l\" (UniqueName: \"kubernetes.io/projected/977d57fe-4c6f-4520-882a-386adc295618-kube-api-access-c8f6l\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.521450 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.521420 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/977d57fe-4c6f-4520-882a-386adc295618-istio-token\") pod \"e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.559512 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.559474 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:09.733026 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.733001 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx"] Jul 10 14:28:09.734882 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:28:09.734850 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977d57fe_4c6f_4520_882a_386adc295618.slice/crio-c6f96e99fb03b2019f4ff5f4aa1dff20f3d8734ac51b7b6ec22b7904354ebdfc WatchSource:0}: Error finding container c6f96e99fb03b2019f4ff5f4aa1dff20f3d8734ac51b7b6ec22b7904354ebdfc: Status 404 returned error can't find the container with id c6f96e99fb03b2019f4ff5f4aa1dff20f3d8734ac51b7b6ec22b7904354ebdfc Jul 10 14:28:09.736964 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.736932 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:28:09.737045 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.737012 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:28:09.737081 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:09.737041 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:28:10.538897 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:10.538860 2576 patch_prober.go:28] interesting pod/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 14:28:10.539390 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:10.538920 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" podUID="118c0028-efd7-45bc-907b-11afad71fec3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 14:28:10.663794 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:10.663755 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" event={"ID":"977d57fe-4c6f-4520-882a-386adc295618","Type":"ContainerStarted","Data":"c347802d5f1fc5cdab75f6eedf9dea18cd03b677ec33c3f29438939978cc2014"} Jul 10 14:28:10.663999 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:10.663845 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" event={"ID":"977d57fe-4c6f-4520-882a-386adc295618","Type":"ContainerStarted","Data":"c6f96e99fb03b2019f4ff5f4aa1dff20f3d8734ac51b7b6ec22b7904354ebdfc"} Jul 10 14:28:10.715693 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:10.715639 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" podStartSLOduration=1.715621431 podStartE2EDuration="1.715621431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:28:10.713054849 +0000 UTC m=+2298.176660295" watchObservedRunningTime="2026-07-10 14:28:10.715621431 +0000 UTC m=+2298.179226877" Jul 10 14:28:11.560370 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:11.560329 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:11.564981 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:11.564953 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:11.669742 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:11.669708 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:11.670630 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:11.670610 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:13.171664 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.171635 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:13.298226 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.298198 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:28:13.402037 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.402002 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:13.430246 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.430212 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-istio-data\") pod \"118c0028-efd7-45bc-907b-11afad71fec3\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " Jul 10 14:28:13.430440 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.430271 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/118c0028-efd7-45bc-907b-11afad71fec3-istio-podinfo\") pod \"118c0028-efd7-45bc-907b-11afad71fec3\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " Jul 10 14:28:13.430440 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.430317 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-workload-socket\") pod \"118c0028-efd7-45bc-907b-11afad71fec3\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " Jul 10 14:28:13.430440 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.430351 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-credential-socket\") pod \"118c0028-efd7-45bc-907b-11afad71fec3\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " Jul 10 14:28:13.430440 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.430372 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/118c0028-efd7-45bc-907b-11afad71fec3-istio-token\") pod \"118c0028-efd7-45bc-907b-11afad71fec3\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " Jul 10 14:28:13.430440 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.430416 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rkcq\" (UniqueName: \"kubernetes.io/projected/118c0028-efd7-45bc-907b-11afad71fec3-kube-api-access-7rkcq\") pod \"118c0028-efd7-45bc-907b-11afad71fec3\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " Jul 10 14:28:13.430697 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.430465 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-workload-certs\") pod \"118c0028-efd7-45bc-907b-11afad71fec3\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " Jul 10 14:28:13.430697 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.430531 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-istio-envoy\") pod \"118c0028-efd7-45bc-907b-11afad71fec3\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " Jul 10 14:28:13.430697 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.430555 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/118c0028-efd7-45bc-907b-11afad71fec3-istiod-ca-cert\") pod \"118c0028-efd7-45bc-907b-11afad71fec3\" (UID: \"118c0028-efd7-45bc-907b-11afad71fec3\") " Jul 10 14:28:13.430697 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.430603 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "118c0028-efd7-45bc-907b-11afad71fec3" (UID: "118c0028-efd7-45bc-907b-11afad71fec3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:28:13.430697 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.430618 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "118c0028-efd7-45bc-907b-11afad71fec3" (UID: "118c0028-efd7-45bc-907b-11afad71fec3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:28:13.430992 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.430827 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "118c0028-efd7-45bc-907b-11afad71fec3" (UID: "118c0028-efd7-45bc-907b-11afad71fec3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:28:13.430992 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.430848 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-workload-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:28:13.430992 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.430867 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-credential-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:28:13.431153 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.431045 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-istio-data" (OuterVolumeSpecName: "istio-data") pod "118c0028-efd7-45bc-907b-11afad71fec3" (UID: "118c0028-efd7-45bc-907b-11afad71fec3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:28:13.431153 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.431114 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/118c0028-efd7-45bc-907b-11afad71fec3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "118c0028-efd7-45bc-907b-11afad71fec3" (UID: "118c0028-efd7-45bc-907b-11afad71fec3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:28:13.432633 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.432604 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/118c0028-efd7-45bc-907b-11afad71fec3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "118c0028-efd7-45bc-907b-11afad71fec3" (UID: "118c0028-efd7-45bc-907b-11afad71fec3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 14:28:13.432754 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.432651 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118c0028-efd7-45bc-907b-11afad71fec3-istio-token" (OuterVolumeSpecName: "istio-token") pod "118c0028-efd7-45bc-907b-11afad71fec3" (UID: "118c0028-efd7-45bc-907b-11afad71fec3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:28:13.432754 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.432676 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118c0028-efd7-45bc-907b-11afad71fec3-kube-api-access-7rkcq" (OuterVolumeSpecName: "kube-api-access-7rkcq") pod "118c0028-efd7-45bc-907b-11afad71fec3" (UID: "118c0028-efd7-45bc-907b-11afad71fec3"). InnerVolumeSpecName "kube-api-access-7rkcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:28:13.432754 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.432728 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "118c0028-efd7-45bc-907b-11afad71fec3" (UID: "118c0028-efd7-45bc-907b-11afad71fec3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:28:13.523868 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.523794 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:13.532139 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.532107 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-istio-data\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:28:13.532139 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.532140 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/118c0028-efd7-45bc-907b-11afad71fec3-istio-podinfo\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:28:13.532318 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.532151 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/118c0028-efd7-45bc-907b-11afad71fec3-istio-token\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:28:13.532318 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.532163 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rkcq\" (UniqueName: \"kubernetes.io/projected/118c0028-efd7-45bc-907b-11afad71fec3-kube-api-access-7rkcq\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:28:13.532318 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.532178 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-workload-certs\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:28:13.532318 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.532188 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/118c0028-efd7-45bc-907b-11afad71fec3-istio-envoy\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:28:13.532318 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.532200 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/118c0028-efd7-45bc-907b-11afad71fec3-istiod-ca-cert\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:28:13.666319 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.666218 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:13.680306 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.680273 2576 generic.go:356] "Generic (PLEG): container finished" podID="118c0028-efd7-45bc-907b-11afad71fec3" containerID="fd43afae6b810fb87762f0b0992022befeb9a2c8742f71945cd487e0440851c5" exitCode=0 Jul 10 14:28:13.680505 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.680336 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" Jul 10 14:28:13.680505 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.680370 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" event={"ID":"118c0028-efd7-45bc-907b-11afad71fec3","Type":"ContainerDied","Data":"fd43afae6b810fb87762f0b0992022befeb9a2c8742f71945cd487e0440851c5"} Jul 10 14:28:13.680505 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.680426 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q" event={"ID":"118c0028-efd7-45bc-907b-11afad71fec3","Type":"ContainerDied","Data":"548c4b7a9cae11a2f9ca0b274867787f009731d97d75c335f7d6b81e050e7d1d"} Jul 10 14:28:13.680505 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.680449 2576 scope.go:117] "RemoveContainer" containerID="fd43afae6b810fb87762f0b0992022befeb9a2c8742f71945cd487e0440851c5" Jul 10 14:28:13.689518 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.689498 2576 scope.go:117] "RemoveContainer" containerID="bdfdbb9b6a074e361e62e6fac1ed52d9782d003e008e4f4b9062c9f4efd384a4" Jul 10 14:28:13.697911 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.697886 2576 scope.go:117] "RemoveContainer" containerID="fd43afae6b810fb87762f0b0992022befeb9a2c8742f71945cd487e0440851c5" Jul 10 14:28:13.698243 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:28:13.698222 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd43afae6b810fb87762f0b0992022befeb9a2c8742f71945cd487e0440851c5\": container with ID starting with fd43afae6b810fb87762f0b0992022befeb9a2c8742f71945cd487e0440851c5 not found: ID does not exist" containerID="fd43afae6b810fb87762f0b0992022befeb9a2c8742f71945cd487e0440851c5" Jul 10 14:28:13.698335 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.698258 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd43afae6b810fb87762f0b0992022befeb9a2c8742f71945cd487e0440851c5"} err="failed to get container status \"fd43afae6b810fb87762f0b0992022befeb9a2c8742f71945cd487e0440851c5\": rpc error: code = NotFound desc = could not find container \"fd43afae6b810fb87762f0b0992022befeb9a2c8742f71945cd487e0440851c5\": container with ID starting with fd43afae6b810fb87762f0b0992022befeb9a2c8742f71945cd487e0440851c5 not found: ID does not exist" Jul 10 14:28:13.698335 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.698286 2576 scope.go:117] "RemoveContainer" containerID="bdfdbb9b6a074e361e62e6fac1ed52d9782d003e008e4f4b9062c9f4efd384a4" Jul 10 14:28:13.698577 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:28:13.698559 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdfdbb9b6a074e361e62e6fac1ed52d9782d003e008e4f4b9062c9f4efd384a4\": container with ID starting with bdfdbb9b6a074e361e62e6fac1ed52d9782d003e008e4f4b9062c9f4efd384a4 not found: ID does not exist" containerID="bdfdbb9b6a074e361e62e6fac1ed52d9782d003e008e4f4b9062c9f4efd384a4" Jul 10 14:28:13.698615 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.698583 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdfdbb9b6a074e361e62e6fac1ed52d9782d003e008e4f4b9062c9f4efd384a4"} err="failed to get container status \"bdfdbb9b6a074e361e62e6fac1ed52d9782d003e008e4f4b9062c9f4efd384a4\": rpc error: code = NotFound desc = could not find container \"bdfdbb9b6a074e361e62e6fac1ed52d9782d003e008e4f4b9062c9f4efd384a4\": container with ID starting with bdfdbb9b6a074e361e62e6fac1ed52d9782d003e008e4f4b9062c9f4efd384a4 not found: ID does not exist" Jul 10 14:28:13.732297 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.732261 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q"] Jul 10 14:28:13.765008 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:13.764969 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e3d9d1-openshift-default-6b7688675f-hd85q"] Jul 10 14:28:14.660047 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:14.660008 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:14.785692 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:14.785653 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:15.037054 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:15.036970 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="118c0028-efd7-45bc-907b-11afad71fec3" path="/var/lib/kubelet/pods/118c0028-efd7-45bc-907b-11afad71fec3/volumes" Jul 10 14:28:17.527833 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:17.527772 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:18.109003 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.108970 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:18.447491 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.447456 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd"] Jul 10 14:28:18.447783 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.447771 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="118c0028-efd7-45bc-907b-11afad71fec3" containerName="istio-proxy" Jul 10 14:28:18.447859 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.447784 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="118c0028-efd7-45bc-907b-11afad71fec3" containerName="istio-proxy" Jul 10 14:28:18.447859 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.447802 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd16bfab-e709-48bc-985c-6b498a3d4020" containerName="istio-proxy" Jul 10 14:28:18.447859 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.447820 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd16bfab-e709-48bc-985c-6b498a3d4020" containerName="istio-proxy" Jul 10 14:28:18.447956 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.447877 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="118c0028-efd7-45bc-907b-11afad71fec3" containerName="istio-proxy" Jul 10 14:28:18.447956 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.447888 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="118c0028-efd7-45bc-907b-11afad71fec3" containerName="istio-proxy" Jul 10 14:28:18.452934 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.452910 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.457540 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.457519 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b5750f-openshift-default-dockercfg-9snw5\"" Jul 10 14:28:18.467726 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.467696 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd"] Jul 10 14:28:18.484611 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.484583 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:18.578535 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.578496 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-workload-socket\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.579048 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.578542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-token\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.579048 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.578615 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-podinfo\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.579048 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.578651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-data\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.579048 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.578680 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istiod-ca-cert\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.579048 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.578735 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-credential-socket\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.579048 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.578764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-workload-certs\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.579048 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.578794 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x7vs\" (UniqueName: \"kubernetes.io/projected/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-kube-api-access-8x7vs\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.579048 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.578882 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-envoy\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.679604 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.679561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-envoy\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.679851 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.679639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-workload-socket\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.679851 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.679662 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-token\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.679851 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.679714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-podinfo\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.679851 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.679752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-data\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.679851 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.679778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istiod-ca-cert\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.679851 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.679841 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-credential-socket\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.680184 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.679964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-workload-certs\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.680184 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.680015 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x7vs\" (UniqueName: \"kubernetes.io/projected/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-kube-api-access-8x7vs\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.680184 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.680158 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-workload-socket\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.680576 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.680553 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-workload-certs\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.680674 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.680632 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istiod-ca-cert\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.680674 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.680660 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-credential-socket\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.680746 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.680664 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-data\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.682243 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.682216 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-podinfo\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.682347 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.682268 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-envoy\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.695057 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.695028 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x7vs\" (UniqueName: \"kubernetes.io/projected/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-kube-api-access-8x7vs\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.695229 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.695209 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-token\") pod \"e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.718717 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.718644 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:28:18.763630 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.763591 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:18.925106 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.924958 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd"] Jul 10 14:28:18.927336 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:28:18.927308 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2e1e6c_4b59_45f3_b283_0ef3032e8790.slice/crio-06dab930915059995469fbc647c837ffb0e05953f7e8355bb74d08f8ebacd0c1 WatchSource:0}: Error finding container 06dab930915059995469fbc647c837ffb0e05953f7e8355bb74d08f8ebacd0c1: Status 404 returned error can't find the container with id 06dab930915059995469fbc647c837ffb0e05953f7e8355bb74d08f8ebacd0c1 Jul 10 14:28:18.929526 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.929494 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:28:18.929628 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.929555 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:28:18.929628 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:18.929582 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:28:19.165406 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:19.165369 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:19.713303 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:19.713264 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" event={"ID":"1b2e1e6c-4b59-45f3-b283-0ef3032e8790","Type":"ContainerStarted","Data":"e81bd4efb3441537c0a24006ae92b71905d52560e28e7fe356e9a8c9e335d536"} Jul 10 14:28:19.713708 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:19.713320 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" event={"ID":"1b2e1e6c-4b59-45f3-b283-0ef3032e8790","Type":"ContainerStarted","Data":"06dab930915059995469fbc647c837ffb0e05953f7e8355bb74d08f8ebacd0c1"} Jul 10 14:28:19.748217 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:19.748157 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" podStartSLOduration=1.7481390860000001 podStartE2EDuration="1.748139086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:28:19.747169917 +0000 UTC m=+2307.210775363" watchObservedRunningTime="2026-07-10 14:28:19.748139086 +0000 UTC m=+2307.211744531" Jul 10 14:28:19.764201 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:19.764165 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:19.769101 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:19.769068 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:20.718128 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:20.718091 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:20.718990 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:20.718970 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:21.983358 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:21.983326 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:22.099031 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:22.098997 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:22.200087 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:22.200055 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:22.301919 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:22.301834 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:22.898156 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:22.898124 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:23.020773 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:23.020739 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:25.582742 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:25.582707 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:26.264096 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:26.264065 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:26.671403 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:26.671369 2576 patch_prober.go:28] interesting pod/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 10 14:28:26.671894 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:26.671437 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" podUID="977d57fe-4c6f-4520-882a-386adc295618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 10 14:28:26.752856 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:26.752791 2576 generic.go:356] "Generic (PLEG): container finished" podID="977d57fe-4c6f-4520-882a-386adc295618" containerID="c347802d5f1fc5cdab75f6eedf9dea18cd03b677ec33c3f29438939978cc2014" exitCode=0 Jul 10 14:28:26.753033 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:26.752865 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" event={"ID":"977d57fe-4c6f-4520-882a-386adc295618","Type":"ContainerDied","Data":"c347802d5f1fc5cdab75f6eedf9dea18cd03b677ec33c3f29438939978cc2014"} Jul 10 14:28:26.753318 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:26.753303 2576 scope.go:117] "RemoveContainer" containerID="c347802d5f1fc5cdab75f6eedf9dea18cd03b677ec33c3f29438939978cc2014" Jul 10 14:28:26.754403 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:26.754374 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:28:26.754472 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:26.754444 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:28:26.754508 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:26.754490 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:28:27.763359 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:27.763318 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" event={"ID":"977d57fe-4c6f-4520-882a-386adc295618","Type":"ContainerStarted","Data":"3e84d0e74eb94692c7f618e7ec652b9240573ce1a652011150898c871dc2a68b"} Jul 10 14:28:28.560885 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:28.560843 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:28.561120 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:28.560989 2576 patch_prober.go:28] interesting pod/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 10 14:28:28.561120 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:28.561034 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" podUID="977d57fe-4c6f-4520-882a-386adc295618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 10 14:28:29.560528 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:29.560481 2576 patch_prober.go:28] interesting pod/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 10 14:28:29.561087 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:29.560577 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" podUID="977d57fe-4c6f-4520-882a-386adc295618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 10 14:28:30.560290 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:30.560246 2576 patch_prober.go:28] interesting pod/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 10 14:28:30.560559 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:30.560528 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" podUID="977d57fe-4c6f-4520-882a-386adc295618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 10 14:28:31.560642 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:31.560593 2576 patch_prober.go:28] interesting pod/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 10 14:28:31.561206 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:31.560717 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" podUID="977d57fe-4c6f-4520-882a-386adc295618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 10 14:28:32.564417 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:32.564386 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:32.564868 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:32.564737 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:32.565617 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:32.565599 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:28:33.540171 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:33.540120 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:28:34.805528 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:34.805443 2576 generic.go:356] "Generic (PLEG): container finished" podID="1b2e1e6c-4b59-45f3-b283-0ef3032e8790" containerID="e81bd4efb3441537c0a24006ae92b71905d52560e28e7fe356e9a8c9e335d536" exitCode=0 Jul 10 14:28:34.805968 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:34.805520 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" event={"ID":"1b2e1e6c-4b59-45f3-b283-0ef3032e8790","Type":"ContainerDied","Data":"e81bd4efb3441537c0a24006ae92b71905d52560e28e7fe356e9a8c9e335d536"} Jul 10 14:28:34.805968 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:34.805929 2576 scope.go:117] "RemoveContainer" containerID="e81bd4efb3441537c0a24006ae92b71905d52560e28e7fe356e9a8c9e335d536" Jul 10 14:28:34.806956 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:34.806931 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:28:34.807009 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:34.806997 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:28:34.807045 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:34.807026 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:28:35.017582 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:35.017544 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:35.119554 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:35.119447 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:35.718318 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:35.718274 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:35.812891 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:35.812856 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" event={"ID":"1b2e1e6c-4b59-45f3-b283-0ef3032e8790","Type":"ContainerStarted","Data":"78b2d42e1bb3467b81dbc967ead0c76533408c274fce780aa266757fb11163aa"} Jul 10 14:28:36.764174 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:36.764129 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:36.765448 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:36.765423 2576 patch_prober.go:28] interesting pod/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 10 14:28:36.765567 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:36.765476 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" podUID="1b2e1e6c-4b59-45f3-b283-0ef3032e8790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 10 14:28:37.764353 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:37.764309 2576 patch_prober.go:28] interesting pod/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 10 14:28:37.764778 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:37.764404 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" podUID="1b2e1e6c-4b59-45f3-b283-0ef3032e8790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 10 14:28:38.763989 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:38.763950 2576 patch_prober.go:28] interesting pod/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 10 14:28:38.764210 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:38.764026 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" podUID="1b2e1e6c-4b59-45f3-b283-0ef3032e8790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 10 14:28:39.764236 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:39.764202 2576 patch_prober.go:28] interesting pod/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 10 14:28:39.764628 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:39.764257 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" podUID="1b2e1e6c-4b59-45f3-b283-0ef3032e8790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 10 14:28:40.768515 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:40.768480 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:40.769011 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:40.768760 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:40.769465 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:40.769450 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:28:43.821850 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:43.821801 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:43.927507 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:43.927470 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:44.628792 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:44.628705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:48.513476 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:48.513426 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:28:49.384668 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:49.384628 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:57.786471 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:57.786430 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:28:57.904792 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:28:57.904755 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:00.085849 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:00.085799 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx"] Jul 10 14:29:00.086371 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:00.086012 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" podUID="977d57fe-4c6f-4520-882a-386adc295618" containerName="istio-proxy" containerID="cri-o://3e84d0e74eb94692c7f618e7ec652b9240573ce1a652011150898c871dc2a68b" gracePeriod=30 Jul 10 14:29:00.113866 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:00.113836 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:00.354878 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:00.354765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:02.567231 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:02.567192 2576 patch_prober.go:28] interesting pod/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 14:29:02.567630 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:02.567265 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" podUID="977d57fe-4c6f-4520-882a-386adc295618" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 14:29:05.331032 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.331003 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:29:05.418981 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.418946 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-workload-socket\") pod \"977d57fe-4c6f-4520-882a-386adc295618\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " Jul 10 14:29:05.419178 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.419011 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/977d57fe-4c6f-4520-882a-386adc295618-istio-podinfo\") pod \"977d57fe-4c6f-4520-882a-386adc295618\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " Jul 10 14:29:05.419178 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.419038 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8f6l\" (UniqueName: \"kubernetes.io/projected/977d57fe-4c6f-4520-882a-386adc295618-kube-api-access-c8f6l\") pod \"977d57fe-4c6f-4520-882a-386adc295618\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " Jul 10 14:29:05.419178 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.419066 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-workload-certs\") pod \"977d57fe-4c6f-4520-882a-386adc295618\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " Jul 10 14:29:05.419178 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.419102 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-credential-socket\") pod \"977d57fe-4c6f-4520-882a-386adc295618\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " Jul 10 14:29:05.419178 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.419116 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/977d57fe-4c6f-4520-882a-386adc295618-istio-token\") pod \"977d57fe-4c6f-4520-882a-386adc295618\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " Jul 10 14:29:05.419450 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.419206 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/977d57fe-4c6f-4520-882a-386adc295618-istiod-ca-cert\") pod \"977d57fe-4c6f-4520-882a-386adc295618\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " Jul 10 14:29:05.419450 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.419244 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-istio-data\") pod \"977d57fe-4c6f-4520-882a-386adc295618\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " Jul 10 14:29:05.419450 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.419268 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-istio-envoy\") pod \"977d57fe-4c6f-4520-882a-386adc295618\" (UID: \"977d57fe-4c6f-4520-882a-386adc295618\") " Jul 10 14:29:05.419450 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.419316 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "977d57fe-4c6f-4520-882a-386adc295618" (UID: "977d57fe-4c6f-4520-882a-386adc295618"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:29:05.419658 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.419459 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "977d57fe-4c6f-4520-882a-386adc295618" (UID: "977d57fe-4c6f-4520-882a-386adc295618"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:29:05.419658 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.419500 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "977d57fe-4c6f-4520-882a-386adc295618" (UID: "977d57fe-4c6f-4520-882a-386adc295618"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:29:05.419658 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.419613 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-workload-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:29:05.419658 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.419614 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/977d57fe-4c6f-4520-882a-386adc295618-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "977d57fe-4c6f-4520-882a-386adc295618" (UID: "977d57fe-4c6f-4520-882a-386adc295618"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:29:05.419658 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.419634 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-workload-certs\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:29:05.419658 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.419649 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-credential-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:29:05.420286 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.420258 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-istio-data" (OuterVolumeSpecName: "istio-data") pod "977d57fe-4c6f-4520-882a-386adc295618" (UID: "977d57fe-4c6f-4520-882a-386adc295618"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:29:05.421452 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.421422 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "977d57fe-4c6f-4520-882a-386adc295618" (UID: "977d57fe-4c6f-4520-882a-386adc295618"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:29:05.421452 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.421441 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977d57fe-4c6f-4520-882a-386adc295618-kube-api-access-c8f6l" (OuterVolumeSpecName: "kube-api-access-c8f6l") pod "977d57fe-4c6f-4520-882a-386adc295618" (UID: "977d57fe-4c6f-4520-882a-386adc295618"). InnerVolumeSpecName "kube-api-access-c8f6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:29:05.421610 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.421499 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977d57fe-4c6f-4520-882a-386adc295618-istio-token" (OuterVolumeSpecName: "istio-token") pod "977d57fe-4c6f-4520-882a-386adc295618" (UID: "977d57fe-4c6f-4520-882a-386adc295618"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:29:05.421610 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.421526 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/977d57fe-4c6f-4520-882a-386adc295618-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "977d57fe-4c6f-4520-882a-386adc295618" (UID: "977d57fe-4c6f-4520-882a-386adc295618"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 14:29:05.520535 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.520508 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/977d57fe-4c6f-4520-882a-386adc295618-istio-podinfo\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:29:05.520535 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.520532 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8f6l\" (UniqueName: \"kubernetes.io/projected/977d57fe-4c6f-4520-882a-386adc295618-kube-api-access-c8f6l\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:29:05.520535 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.520542 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/977d57fe-4c6f-4520-882a-386adc295618-istio-token\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:29:05.520889 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.520551 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/977d57fe-4c6f-4520-882a-386adc295618-istiod-ca-cert\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:29:05.520889 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.520559 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-istio-data\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:29:05.520889 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.520566 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/977d57fe-4c6f-4520-882a-386adc295618-istio-envoy\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:29:05.976083 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.976050 2576 generic.go:356] "Generic (PLEG): container finished" podID="977d57fe-4c6f-4520-882a-386adc295618" containerID="3e84d0e74eb94692c7f618e7ec652b9240573ce1a652011150898c871dc2a68b" exitCode=0 Jul 10 14:29:05.976288 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.976100 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" Jul 10 14:29:05.976288 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.976133 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" event={"ID":"977d57fe-4c6f-4520-882a-386adc295618","Type":"ContainerDied","Data":"3e84d0e74eb94692c7f618e7ec652b9240573ce1a652011150898c871dc2a68b"} Jul 10 14:29:05.976288 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.976196 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx" event={"ID":"977d57fe-4c6f-4520-882a-386adc295618","Type":"ContainerDied","Data":"c6f96e99fb03b2019f4ff5f4aa1dff20f3d8734ac51b7b6ec22b7904354ebdfc"} Jul 10 14:29:05.976288 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.976228 2576 scope.go:117] "RemoveContainer" containerID="3e84d0e74eb94692c7f618e7ec652b9240573ce1a652011150898c871dc2a68b" Jul 10 14:29:05.986149 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.986128 2576 scope.go:117] "RemoveContainer" containerID="c347802d5f1fc5cdab75f6eedf9dea18cd03b677ec33c3f29438939978cc2014" Jul 10 14:29:05.994722 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.994698 2576 scope.go:117] "RemoveContainer" containerID="3e84d0e74eb94692c7f618e7ec652b9240573ce1a652011150898c871dc2a68b" Jul 10 14:29:05.995032 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:29:05.995008 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e84d0e74eb94692c7f618e7ec652b9240573ce1a652011150898c871dc2a68b\": container with ID starting with 3e84d0e74eb94692c7f618e7ec652b9240573ce1a652011150898c871dc2a68b not found: ID does not exist" containerID="3e84d0e74eb94692c7f618e7ec652b9240573ce1a652011150898c871dc2a68b" Jul 10 14:29:05.995091 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.995042 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e84d0e74eb94692c7f618e7ec652b9240573ce1a652011150898c871dc2a68b"} err="failed to get container status \"3e84d0e74eb94692c7f618e7ec652b9240573ce1a652011150898c871dc2a68b\": rpc error: code = NotFound desc = could not find container \"3e84d0e74eb94692c7f618e7ec652b9240573ce1a652011150898c871dc2a68b\": container with ID starting with 3e84d0e74eb94692c7f618e7ec652b9240573ce1a652011150898c871dc2a68b not found: ID does not exist" Jul 10 14:29:05.995091 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.995061 2576 scope.go:117] "RemoveContainer" containerID="c347802d5f1fc5cdab75f6eedf9dea18cd03b677ec33c3f29438939978cc2014" Jul 10 14:29:05.995280 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:29:05.995264 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c347802d5f1fc5cdab75f6eedf9dea18cd03b677ec33c3f29438939978cc2014\": container with ID starting with c347802d5f1fc5cdab75f6eedf9dea18cd03b677ec33c3f29438939978cc2014 not found: ID does not exist" containerID="c347802d5f1fc5cdab75f6eedf9dea18cd03b677ec33c3f29438939978cc2014" Jul 10 14:29:05.995327 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:05.995285 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c347802d5f1fc5cdab75f6eedf9dea18cd03b677ec33c3f29438939978cc2014"} err="failed to get container status \"c347802d5f1fc5cdab75f6eedf9dea18cd03b677ec33c3f29438939978cc2014\": rpc error: code = NotFound desc = could not find container \"c347802d5f1fc5cdab75f6eedf9dea18cd03b677ec33c3f29438939978cc2014\": container with ID starting with c347802d5f1fc5cdab75f6eedf9dea18cd03b677ec33c3f29438939978cc2014 not found: ID does not exist" Jul 10 14:29:06.002949 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:06.002919 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx"] Jul 10 14:29:06.008900 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:06.008866 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7beb38-openshift-default-6498687cd9-2nhnx"] Jul 10 14:29:07.036102 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:07.036069 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="977d57fe-4c6f-4520-882a-386adc295618" path="/var/lib/kubelet/pods/977d57fe-4c6f-4520-882a-386adc295618/volumes" Jul 10 14:29:11.083925 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:11.083892 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:14.434915 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:14.434876 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:29:15.571090 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:15.571056 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:17.991035 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:17.990998 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:18.091201 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:18.091170 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:22.801772 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:22.801722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:22.892537 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:22.892499 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:26.246537 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:26.246505 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd"] Jul 10 14:29:26.247051 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:26.246705 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" podUID="1b2e1e6c-4b59-45f3-b283-0ef3032e8790" containerName="istio-proxy" containerID="cri-o://78b2d42e1bb3467b81dbc967ead0c76533408c274fce780aa266757fb11163aa" gracePeriod=30 Jul 10 14:29:26.297882 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:26.297842 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:26.565368 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:26.565274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:26.986324 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:26.986290 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:27.192509 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.192478 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx"] Jul 10 14:29:27.192847 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.192833 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="118c0028-efd7-45bc-907b-11afad71fec3" containerName="istio-proxy" Jul 10 14:29:27.192847 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.192847 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="118c0028-efd7-45bc-907b-11afad71fec3" containerName="istio-proxy" Jul 10 14:29:27.192949 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.192858 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977d57fe-4c6f-4520-882a-386adc295618" containerName="istio-proxy" Jul 10 14:29:27.192949 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.192863 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="977d57fe-4c6f-4520-882a-386adc295618" containerName="istio-proxy" Jul 10 14:29:27.192949 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.192925 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="977d57fe-4c6f-4520-882a-386adc295618" containerName="istio-proxy" Jul 10 14:29:27.197379 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.197356 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.200867 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.200845 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-e5410e-openshift-default-dockercfg-24d88\"" Jul 10 14:29:27.212179 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.212033 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m746w\" (UniqueName: \"kubernetes.io/projected/4ebf6590-457f-4b2f-b253-e805f17c409f-kube-api-access-m746w\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.212491 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.212462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-data\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.212619 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.212548 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-workload-certs\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.212680 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.212628 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-workload-socket\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.212680 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.212674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-token\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.212782 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.212727 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-envoy\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.212782 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.212771 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-podinfo\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.212923 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.212842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebf6590-457f-4b2f-b253-e805f17c409f-istiod-ca-cert\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.212923 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.212893 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-credential-socket\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.223857 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.223827 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx"] Jul 10 14:29:27.270161 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.270052 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:27.313890 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.313848 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-workload-certs\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.314095 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.313922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-workload-socket\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.314095 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.313964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-token\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.314095 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.314011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-envoy\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.314095 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.314049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-podinfo\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.314326 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.314097 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebf6590-457f-4b2f-b253-e805f17c409f-istiod-ca-cert\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.314326 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.314138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-credential-socket\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.314326 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.314184 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m746w\" (UniqueName: \"kubernetes.io/projected/4ebf6590-457f-4b2f-b253-e805f17c409f-kube-api-access-m746w\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.314326 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.314212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-data\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.314538 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.314318 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-workload-socket\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.314764 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.314738 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-workload-certs\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.315053 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.315023 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebf6590-457f-4b2f-b253-e805f17c409f-istiod-ca-cert\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.315263 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.315241 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-credential-socket\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.315339 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.315316 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-data\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.317041 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.317019 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-podinfo\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.317209 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.317191 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-envoy\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.322836 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.322798 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-token\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.323065 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.323038 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m746w\" (UniqueName: \"kubernetes.io/projected/4ebf6590-457f-4b2f-b253-e805f17c409f-kube-api-access-m746w\") pod \"e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.466216 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.466170 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:27.507672 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.507637 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:27.649467 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.649445 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx"] Jul 10 14:29:27.651472 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:29:27.651445 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebf6590_457f_4b2f_b253_e805f17c409f.slice/crio-0b929df7e96e7a60a7f3f2b9433efe62582593db798afed7b221884ff9fc2ec1 WatchSource:0}: Error finding container 0b929df7e96e7a60a7f3f2b9433efe62582593db798afed7b221884ff9fc2ec1: Status 404 returned error can't find the container with id 0b929df7e96e7a60a7f3f2b9433efe62582593db798afed7b221884ff9fc2ec1 Jul 10 14:29:27.653789 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.653756 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:29:27.653890 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.653847 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:29:27.653890 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:27.653876 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:29:28.081374 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:28.081328 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" event={"ID":"4ebf6590-457f-4b2f-b253-e805f17c409f","Type":"ContainerStarted","Data":"ae2db71d417029ded22b2f71dae7a987ef5d54da54da085384cfdc74dfb6b21c"} Jul 10 14:29:28.081638 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:28.081398 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" event={"ID":"4ebf6590-457f-4b2f-b253-e805f17c409f","Type":"ContainerStarted","Data":"0b929df7e96e7a60a7f3f2b9433efe62582593db798afed7b221884ff9fc2ec1"} Jul 10 14:29:28.105914 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:28.105859 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" podStartSLOduration=1.105842211 podStartE2EDuration="1.105842211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:29:28.103348459 +0000 UTC m=+2375.566953905" watchObservedRunningTime="2026-07-10 14:29:28.105842211 +0000 UTC m=+2375.569447656" Jul 10 14:29:28.508264 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:28.508230 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:28.513079 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:28.513050 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:29.086825 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:29.086778 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:29.087667 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:29.087649 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:30.097551 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:30.097519 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:30.244102 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:30.244068 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:30.415172 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:30.415131 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:30.570001 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:30.569965 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:30.928223 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:30.928187 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:31.096090 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.096053 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:31.495791 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.495765 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:29:31.553278 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.553184 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-token\") pod \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " Jul 10 14:29:31.553278 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.553270 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-podinfo\") pod \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " Jul 10 14:29:31.553497 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.553299 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8x7vs\" (UniqueName: \"kubernetes.io/projected/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-kube-api-access-8x7vs\") pod \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " Jul 10 14:29:31.553497 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.553322 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-envoy\") pod \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " Jul 10 14:29:31.553497 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.553351 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-workload-socket\") pod \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " Jul 10 14:29:31.553497 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.553376 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istiod-ca-cert\") pod \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " Jul 10 14:29:31.553497 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.553399 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-workload-certs\") pod \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " Jul 10 14:29:31.553497 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.553457 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-data\") pod \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " Jul 10 14:29:31.553497 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.553499 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-credential-socket\") pod \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\" (UID: \"1b2e1e6c-4b59-45f3-b283-0ef3032e8790\") " Jul 10 14:29:31.557922 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.554675 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1b2e1e6c-4b59-45f3-b283-0ef3032e8790" (UID: "1b2e1e6c-4b59-45f3-b283-0ef3032e8790"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:29:31.557922 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.554772 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-data" (OuterVolumeSpecName: "istio-data") pod "1b2e1e6c-4b59-45f3-b283-0ef3032e8790" (UID: "1b2e1e6c-4b59-45f3-b283-0ef3032e8790"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:29:31.557922 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.554953 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1b2e1e6c-4b59-45f3-b283-0ef3032e8790" (UID: "1b2e1e6c-4b59-45f3-b283-0ef3032e8790"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:29:31.557922 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.555262 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1b2e1e6c-4b59-45f3-b283-0ef3032e8790" (UID: "1b2e1e6c-4b59-45f3-b283-0ef3032e8790"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:29:31.557922 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.556275 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1b2e1e6c-4b59-45f3-b283-0ef3032e8790" (UID: "1b2e1e6c-4b59-45f3-b283-0ef3032e8790"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 14:29:31.557922 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.556278 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-kube-api-access-8x7vs" (OuterVolumeSpecName: "kube-api-access-8x7vs") pod "1b2e1e6c-4b59-45f3-b283-0ef3032e8790" (UID: "1b2e1e6c-4b59-45f3-b283-0ef3032e8790"). InnerVolumeSpecName "kube-api-access-8x7vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:29:31.557922 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.556574 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1b2e1e6c-4b59-45f3-b283-0ef3032e8790" (UID: "1b2e1e6c-4b59-45f3-b283-0ef3032e8790"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:29:31.562664 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.562629 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-token" (OuterVolumeSpecName: "istio-token") pod "1b2e1e6c-4b59-45f3-b283-0ef3032e8790" (UID: "1b2e1e6c-4b59-45f3-b283-0ef3032e8790"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:29:31.564553 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.564525 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1b2e1e6c-4b59-45f3-b283-0ef3032e8790" (UID: "1b2e1e6c-4b59-45f3-b283-0ef3032e8790"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:29:31.655208 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.655161 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-podinfo\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:29:31.655208 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.655199 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x7vs\" (UniqueName: \"kubernetes.io/projected/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-kube-api-access-8x7vs\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:29:31.655208 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.655211 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-envoy\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:29:31.655208 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.655220 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-workload-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:29:31.655478 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.655228 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istiod-ca-cert\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:29:31.655478 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.655236 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-workload-certs\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:29:31.655478 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.655243 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-data\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:29:31.655478 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.655250 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-credential-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:29:31.655478 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:31.655257 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2e1e6c-4b59-45f3-b283-0ef3032e8790-istio-token\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:29:32.105634 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:32.105599 2576 generic.go:356] "Generic (PLEG): container finished" podID="1b2e1e6c-4b59-45f3-b283-0ef3032e8790" containerID="78b2d42e1bb3467b81dbc967ead0c76533408c274fce780aa266757fb11163aa" exitCode=0 Jul 10 14:29:32.105833 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:32.105661 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" Jul 10 14:29:32.105833 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:32.105688 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" event={"ID":"1b2e1e6c-4b59-45f3-b283-0ef3032e8790","Type":"ContainerDied","Data":"78b2d42e1bb3467b81dbc967ead0c76533408c274fce780aa266757fb11163aa"} Jul 10 14:29:32.105833 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:32.105756 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd" event={"ID":"1b2e1e6c-4b59-45f3-b283-0ef3032e8790","Type":"ContainerDied","Data":"06dab930915059995469fbc647c837ffb0e05953f7e8355bb74d08f8ebacd0c1"} Jul 10 14:29:32.105833 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:32.105786 2576 scope.go:117] "RemoveContainer" containerID="78b2d42e1bb3467b81dbc967ead0c76533408c274fce780aa266757fb11163aa" Jul 10 14:29:32.115174 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:32.115150 2576 scope.go:117] "RemoveContainer" containerID="e81bd4efb3441537c0a24006ae92b71905d52560e28e7fe356e9a8c9e335d536" Jul 10 14:29:32.123613 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:32.123595 2576 scope.go:117] "RemoveContainer" containerID="78b2d42e1bb3467b81dbc967ead0c76533408c274fce780aa266757fb11163aa" Jul 10 14:29:32.123923 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:29:32.123903 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b2d42e1bb3467b81dbc967ead0c76533408c274fce780aa266757fb11163aa\": container with ID starting with 78b2d42e1bb3467b81dbc967ead0c76533408c274fce780aa266757fb11163aa not found: ID does not exist" containerID="78b2d42e1bb3467b81dbc967ead0c76533408c274fce780aa266757fb11163aa" Jul 10 14:29:32.123968 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:32.123932 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b2d42e1bb3467b81dbc967ead0c76533408c274fce780aa266757fb11163aa"} err="failed to get container status \"78b2d42e1bb3467b81dbc967ead0c76533408c274fce780aa266757fb11163aa\": rpc error: code = NotFound desc = could not find container \"78b2d42e1bb3467b81dbc967ead0c76533408c274fce780aa266757fb11163aa\": container with ID starting with 78b2d42e1bb3467b81dbc967ead0c76533408c274fce780aa266757fb11163aa not found: ID does not exist" Jul 10 14:29:32.123968 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:32.123952 2576 scope.go:117] "RemoveContainer" containerID="e81bd4efb3441537c0a24006ae92b71905d52560e28e7fe356e9a8c9e335d536" Jul 10 14:29:32.124206 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:29:32.124184 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e81bd4efb3441537c0a24006ae92b71905d52560e28e7fe356e9a8c9e335d536\": container with ID starting with e81bd4efb3441537c0a24006ae92b71905d52560e28e7fe356e9a8c9e335d536 not found: ID does not exist" containerID="e81bd4efb3441537c0a24006ae92b71905d52560e28e7fe356e9a8c9e335d536" Jul 10 14:29:32.124240 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:32.124214 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e81bd4efb3441537c0a24006ae92b71905d52560e28e7fe356e9a8c9e335d536"} err="failed to get container status \"e81bd4efb3441537c0a24006ae92b71905d52560e28e7fe356e9a8c9e335d536\": rpc error: code = NotFound desc = could not find container \"e81bd4efb3441537c0a24006ae92b71905d52560e28e7fe356e9a8c9e335d536\": container with ID starting with e81bd4efb3441537c0a24006ae92b71905d52560e28e7fe356e9a8c9e335d536 not found: ID does not exist" Jul 10 14:29:32.132998 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:32.132972 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd"] Jul 10 14:29:32.143421 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:32.143395 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b5750f-openshift-default-8579cc4b6c-vm2cd"] Jul 10 14:29:33.036108 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:33.036076 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b2e1e6c-4b59-45f3-b283-0ef3032e8790" path="/var/lib/kubelet/pods/1b2e1e6c-4b59-45f3-b283-0ef3032e8790/volumes" Jul 10 14:29:34.564136 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:34.564101 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:34.868153 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:34.868065 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:35.154046 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.154016 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw"] Jul 10 14:29:35.154367 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.154353 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b2e1e6c-4b59-45f3-b283-0ef3032e8790" containerName="istio-proxy" Jul 10 14:29:35.154408 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.154369 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b2e1e6c-4b59-45f3-b283-0ef3032e8790" containerName="istio-proxy" Jul 10 14:29:35.154408 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.154391 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977d57fe-4c6f-4520-882a-386adc295618" containerName="istio-proxy" Jul 10 14:29:35.154408 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.154396 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="977d57fe-4c6f-4520-882a-386adc295618" containerName="istio-proxy" Jul 10 14:29:35.154512 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.154456 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="977d57fe-4c6f-4520-882a-386adc295618" containerName="istio-proxy" Jul 10 14:29:35.154512 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.154467 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b2e1e6c-4b59-45f3-b283-0ef3032e8790" containerName="istio-proxy" Jul 10 14:29:35.158969 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.158951 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.161779 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.161757 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-47e604-openshift-default-dockercfg-xfbwc\"" Jul 10 14:29:35.173854 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.173827 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw"] Jul 10 14:29:35.187308 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.187267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-credential-socket\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.187465 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.187331 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-workload-socket\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.187465 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.187361 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-token\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.187538 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.187465 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-workload-certs\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.187538 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.187501 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-envoy\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.187639 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.187594 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-data\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.187639 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.187624 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2tkq\" (UniqueName: \"kubernetes.io/projected/35d80e01-aa88-4182-bdbd-e1092352bc83-kube-api-access-v2tkq\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.187751 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.187658 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d80e01-aa88-4182-bdbd-e1092352bc83-istiod-ca-cert\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.187751 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.187716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-podinfo\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.207772 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.207725 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:35.288893 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.288855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-workload-socket\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.288893 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.288897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-token\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.289135 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.288976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-workload-certs\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.289135 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.289100 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-envoy\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.289245 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.289192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-data\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.289245 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.289223 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tkq\" (UniqueName: \"kubernetes.io/projected/35d80e01-aa88-4182-bdbd-e1092352bc83-kube-api-access-v2tkq\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.289349 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.289307 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-workload-socket\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.289349 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.289319 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-workload-certs\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.289349 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.289304 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d80e01-aa88-4182-bdbd-e1092352bc83-istiod-ca-cert\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.289496 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.289383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-podinfo\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.289496 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.289439 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-credential-socket\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.289591 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.289532 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-data\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.289794 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.289771 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-credential-socket\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.289993 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.289971 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d80e01-aa88-4182-bdbd-e1092352bc83-istiod-ca-cert\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.291366 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.291342 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-envoy\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.291793 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.291775 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-podinfo\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.297995 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.297971 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-token\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.299742 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.299718 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2tkq\" (UniqueName: \"kubernetes.io/projected/35d80e01-aa88-4182-bdbd-e1092352bc83-kube-api-access-v2tkq\") pod \"e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.469082 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.468991 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:35.473839 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.473800 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:35.620861 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.620833 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw"] Jul 10 14:29:35.622644 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:29:35.622617 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d80e01_aa88_4182_bdbd_e1092352bc83.slice/crio-788cc84ae724e107d16484937cc9a8703cf24911ab8908fabab8069fe9c196e5 WatchSource:0}: Error finding container 788cc84ae724e107d16484937cc9a8703cf24911ab8908fabab8069fe9c196e5: Status 404 returned error can't find the container with id 788cc84ae724e107d16484937cc9a8703cf24911ab8908fabab8069fe9c196e5 Jul 10 14:29:35.624826 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.624775 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:29:35.624908 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.624861 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:29:35.624908 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:35.624889 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:29:36.126766 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:36.126727 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" event={"ID":"35d80e01-aa88-4182-bdbd-e1092352bc83","Type":"ContainerStarted","Data":"9e0b60c0cee1be2e5a3f57eec0f293098ea53ceb3f192213fedf5e40a911faf0"} Jul 10 14:29:36.126978 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:36.126782 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" event={"ID":"35d80e01-aa88-4182-bdbd-e1092352bc83","Type":"ContainerStarted","Data":"788cc84ae724e107d16484937cc9a8703cf24911ab8908fabab8069fe9c196e5"} Jul 10 14:29:36.155767 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:36.155717 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" podStartSLOduration=1.155699453 podStartE2EDuration="1.155699453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:29:36.153183733 +0000 UTC m=+2383.616789180" watchObservedRunningTime="2026-07-10 14:29:36.155699453 +0000 UTC m=+2383.619304900" Jul 10 14:29:36.204587 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:36.204544 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:29:36.469880 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:36.469843 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:36.474676 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:36.474653 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:37.131952 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:37.131914 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:37.132752 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:37.132736 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:38.187010 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:38.186958 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:39.420134 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:39.420103 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:42.604465 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:42.604413 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:44.272881 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:44.271272 2576 patch_prober.go:28] interesting pod/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:33402->10.133.0.51:15021: read: connection reset by peer" start-of-body= Jul 10 14:29:44.272881 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:44.271358 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" podUID="4ebf6590-457f-4b2f-b253-e805f17c409f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:33402->10.133.0.51:15021: read: connection reset by peer" Jul 10 14:29:45.184520 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:45.184483 2576 generic.go:356] "Generic (PLEG): container finished" podID="4ebf6590-457f-4b2f-b253-e805f17c409f" containerID="ae2db71d417029ded22b2f71dae7a987ef5d54da54da085384cfdc74dfb6b21c" exitCode=0 Jul 10 14:29:45.184520 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:45.184524 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" event={"ID":"4ebf6590-457f-4b2f-b253-e805f17c409f","Type":"ContainerDied","Data":"ae2db71d417029ded22b2f71dae7a987ef5d54da54da085384cfdc74dfb6b21c"} Jul 10 14:29:45.184958 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:45.184943 2576 scope.go:117] "RemoveContainer" containerID="ae2db71d417029ded22b2f71dae7a987ef5d54da54da085384cfdc74dfb6b21c" Jul 10 14:29:45.185966 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:45.185940 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:29:45.186073 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:45.186007 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:29:45.186073 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:45.186037 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:29:46.191726 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:46.191690 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" event={"ID":"4ebf6590-457f-4b2f-b253-e805f17c409f","Type":"ContainerStarted","Data":"706357b6b8aa0ea0984816800c88a94aec18cf4fe4eb45557e22286eb2c7a693"} Jul 10 14:29:46.508450 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:46.508352 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:46.508450 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:46.508393 2576 patch_prober.go:28] interesting pod/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 10 14:29:46.508450 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:46.508439 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" podUID="4ebf6590-457f-4b2f-b253-e805f17c409f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 10 14:29:47.509203 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:47.509163 2576 patch_prober.go:28] interesting pod/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 10 14:29:47.509636 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:47.509231 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" podUID="4ebf6590-457f-4b2f-b253-e805f17c409f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 10 14:29:48.509210 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:48.509164 2576 patch_prober.go:28] interesting pod/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 10 14:29:48.509605 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:48.509255 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" podUID="4ebf6590-457f-4b2f-b253-e805f17c409f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 10 14:29:48.560994 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:48.560952 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:29:49.508674 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:49.508638 2576 patch_prober.go:28] interesting pod/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 10 14:29:49.508903 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:49.508707 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" podUID="4ebf6590-457f-4b2f-b253-e805f17c409f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 10 14:29:50.512280 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:50.512199 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:50.512710 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:50.512582 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:50.513196 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:50.513180 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:29:51.218771 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:51.218738 2576 generic.go:356] "Generic (PLEG): container finished" podID="35d80e01-aa88-4182-bdbd-e1092352bc83" containerID="9e0b60c0cee1be2e5a3f57eec0f293098ea53ceb3f192213fedf5e40a911faf0" exitCode=0 Jul 10 14:29:51.218969 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:51.218832 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" event={"ID":"35d80e01-aa88-4182-bdbd-e1092352bc83","Type":"ContainerDied","Data":"9e0b60c0cee1be2e5a3f57eec0f293098ea53ceb3f192213fedf5e40a911faf0"} Jul 10 14:29:51.219343 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:51.219331 2576 scope.go:117] "RemoveContainer" containerID="9e0b60c0cee1be2e5a3f57eec0f293098ea53ceb3f192213fedf5e40a911faf0" Jul 10 14:29:51.220285 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:51.220263 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:29:51.220334 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:51.220323 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:29:51.220370 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:51.220353 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 14:29:52.132486 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:52.132443 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:52.244986 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:52.244943 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" event={"ID":"35d80e01-aa88-4182-bdbd-e1092352bc83","Type":"ContainerStarted","Data":"363a4fea357671e16d4efa301dff561242094064d4950b34a6a40c6373b24b39"} Jul 10 14:29:52.279146 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:52.279108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:52.469609 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:52.469577 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:52.470580 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:52.470553 2576 patch_prober.go:28] interesting pod/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 10 14:29:52.470692 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:52.470613 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" podUID="35d80e01-aa88-4182-bdbd-e1092352bc83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 10 14:29:53.154073 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:53.154041 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 14:29:53.161042 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:53.160999 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 14:29:53.380465 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:53.380435 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:53.469657 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:53.469555 2576 patch_prober.go:28] interesting pod/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 10 14:29:53.469657 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:53.469617 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" podUID="35d80e01-aa88-4182-bdbd-e1092352bc83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 10 14:29:53.487601 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:53.487566 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:29:54.470485 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:54.470446 2576 patch_prober.go:28] interesting pod/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 10 14:29:54.470910 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:54.470507 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" podUID="35d80e01-aa88-4182-bdbd-e1092352bc83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 10 14:29:55.469736 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:55.469694 2576 patch_prober.go:28] interesting pod/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 10 14:29:55.469950 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:55.469766 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" podUID="35d80e01-aa88-4182-bdbd-e1092352bc83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 10 14:29:56.474501 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:56.474464 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:56.474903 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:56.474757 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:56.475570 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:56.475554 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:29:57.510115 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:29:57.510081 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:30:01.236981 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:01.236949 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:30:01.339956 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:01.339921 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:30:08.621017 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.620982 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl"] Jul 10 14:30:08.621403 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.621343 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b2e1e6c-4b59-45f3-b283-0ef3032e8790" containerName="istio-proxy" Jul 10 14:30:08.621403 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.621354 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b2e1e6c-4b59-45f3-b283-0ef3032e8790" containerName="istio-proxy" Jul 10 14:30:08.621480 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.621417 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b2e1e6c-4b59-45f3-b283-0ef3032e8790" containerName="istio-proxy" Jul 10 14:30:08.624339 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.624322 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.627330 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.627305 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-47e604\"/\"openshift-service-ca.crt\"" Jul 10 14:30:08.628422 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.628400 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-47e604\"/\"default-dockercfg-96rhq\"" Jul 10 14:30:08.628506 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.628441 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-47e604\"/\"kube-root-ca.crt\"" Jul 10 14:30:08.628506 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.628496 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-47e604\"/\"auth-test-model-47e604-kserve-self-signed-certs\"" Jul 10 14:30:08.639187 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.639159 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl"] Jul 10 14:30:08.653297 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.653270 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:30:08.820038 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.819987 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-model-cache\") pod \"auth-test-model-47e604-kserve-5677bf6875-dnsjl\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.820038 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.820043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjtrs\" (UniqueName: \"kubernetes.io/projected/e8975433-5603-4ad2-9840-492ef15f2e65-kube-api-access-gjtrs\") pod \"auth-test-model-47e604-kserve-5677bf6875-dnsjl\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.820316 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.820170 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-home\") pod \"auth-test-model-47e604-kserve-5677bf6875-dnsjl\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.820316 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.820276 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-dshm\") pod \"auth-test-model-47e604-kserve-5677bf6875-dnsjl\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.820423 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.820325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-tmp-dir\") pod \"auth-test-model-47e604-kserve-5677bf6875-dnsjl\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.820423 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.820350 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8975433-5603-4ad2-9840-492ef15f2e65-tls-certs\") pod \"auth-test-model-47e604-kserve-5677bf6875-dnsjl\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.921837 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.921787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-model-cache\") pod \"auth-test-model-47e604-kserve-5677bf6875-dnsjl\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.921837 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.921839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjtrs\" (UniqueName: \"kubernetes.io/projected/e8975433-5603-4ad2-9840-492ef15f2e65-kube-api-access-gjtrs\") pod \"auth-test-model-47e604-kserve-5677bf6875-dnsjl\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.922060 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.921877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-home\") pod \"auth-test-model-47e604-kserve-5677bf6875-dnsjl\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.922060 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.921929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-dshm\") pod \"auth-test-model-47e604-kserve-5677bf6875-dnsjl\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.922138 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.922053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-tmp-dir\") pod \"auth-test-model-47e604-kserve-5677bf6875-dnsjl\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.922138 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.922085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8975433-5603-4ad2-9840-492ef15f2e65-tls-certs\") pod \"auth-test-model-47e604-kserve-5677bf6875-dnsjl\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.922289 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.922265 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-model-cache\") pod \"auth-test-model-47e604-kserve-5677bf6875-dnsjl\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.922377 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.922275 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-home\") pod \"auth-test-model-47e604-kserve-5677bf6875-dnsjl\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.922419 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.922409 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-tmp-dir\") pod \"auth-test-model-47e604-kserve-5677bf6875-dnsjl\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.924174 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.924148 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-dshm\") pod \"auth-test-model-47e604-kserve-5677bf6875-dnsjl\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.924411 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.924395 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8975433-5603-4ad2-9840-492ef15f2e65-tls-certs\") pod \"auth-test-model-47e604-kserve-5677bf6875-dnsjl\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.932132 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.932108 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjtrs\" (UniqueName: \"kubernetes.io/projected/e8975433-5603-4ad2-9840-492ef15f2e65-kube-api-access-gjtrs\") pod \"auth-test-model-47e604-kserve-5677bf6875-dnsjl\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:08.933921 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:08.933895 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:09.081480 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:09.081455 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl"] Jul 10 14:30:09.082700 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:30:09.082668 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8975433_5603_4ad2_9840_492ef15f2e65.slice/crio-e934bc4c472a0fe5dd956886497cc3974ef340864b8b00841378399eb1be52be WatchSource:0}: Error finding container e934bc4c472a0fe5dd956886497cc3974ef340864b8b00841378399eb1be52be: Status 404 returned error can't find the container with id e934bc4c472a0fe5dd956886497cc3974ef340864b8b00841378399eb1be52be Jul 10 14:30:09.332718 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:09.332617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" event={"ID":"e8975433-5603-4ad2-9840-492ef15f2e65","Type":"ContainerStarted","Data":"305e81d29df0d2c98b22a424126631831ad99e9aeb35e2b00f32f74620300109"} Jul 10 14:30:09.332718 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:09.332670 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" event={"ID":"e8975433-5603-4ad2-9840-492ef15f2e65","Type":"ContainerStarted","Data":"e934bc4c472a0fe5dd956886497cc3974ef340864b8b00841378399eb1be52be"} Jul 10 14:30:09.358283 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:09.358226 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" podStartSLOduration=1.35820904 podStartE2EDuration="1.35820904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:30:09.355905109 +0000 UTC m=+2416.819510567" watchObservedRunningTime="2026-07-10 14:30:09.35820904 +0000 UTC m=+2416.821814515" Jul 10 14:30:12.709820 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:12.709771 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:30:18.934505 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:18.934410 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:18.934505 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:18.934469 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:18.947302 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:18.947277 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:19.396902 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:19.396875 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:30:26.216031 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:26.215994 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:30:27.371730 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:27.371693 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:30:44.216845 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:44.216793 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:30:59.244230 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:30:59.244180 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:31:04.679802 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:31:04.679763 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:31:17.329133 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:31:17.329095 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:31:32.018090 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:31:32.018049 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:31:48.735419 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:31:48.735384 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:32:03.620185 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:03.620150 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:32:16.161346 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:16.161305 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:32:17.002047 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:17.002007 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl"] Jul 10 14:32:17.035705 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:17.035664 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:32:22.400001 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:22.399960 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:32:24.846322 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:24.846275 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:32:32.022189 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.022144 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" podUID="e8975433-5603-4ad2-9840-492ef15f2e65" containerName="main" containerID="cri-o://305e81d29df0d2c98b22a424126631831ad99e9aeb35e2b00f32f74620300109" gracePeriod=45 Jul 10 14:32:32.273235 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.273166 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:32:32.334085 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.334047 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-tmp-dir\") pod \"e8975433-5603-4ad2-9840-492ef15f2e65\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " Jul 10 14:32:32.334085 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.334089 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-home\") pod \"e8975433-5603-4ad2-9840-492ef15f2e65\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " Jul 10 14:32:32.334343 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.334131 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjtrs\" (UniqueName: \"kubernetes.io/projected/e8975433-5603-4ad2-9840-492ef15f2e65-kube-api-access-gjtrs\") pod \"e8975433-5603-4ad2-9840-492ef15f2e65\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " Jul 10 14:32:32.334343 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.334180 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-model-cache\") pod \"e8975433-5603-4ad2-9840-492ef15f2e65\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " Jul 10 14:32:32.334343 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.334226 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8975433-5603-4ad2-9840-492ef15f2e65-tls-certs\") pod \"e8975433-5603-4ad2-9840-492ef15f2e65\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " Jul 10 14:32:32.334343 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.334253 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-dshm\") pod \"e8975433-5603-4ad2-9840-492ef15f2e65\" (UID: \"e8975433-5603-4ad2-9840-492ef15f2e65\") " Jul 10 14:32:32.334508 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.334346 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8975433-5603-4ad2-9840-492ef15f2e65" (UID: "e8975433-5603-4ad2-9840-492ef15f2e65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:32:32.334508 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.334399 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-home" (OuterVolumeSpecName: "home") pod "e8975433-5603-4ad2-9840-492ef15f2e65" (UID: "e8975433-5603-4ad2-9840-492ef15f2e65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:32:32.334508 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.334489 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8975433-5603-4ad2-9840-492ef15f2e65" (UID: "e8975433-5603-4ad2-9840-492ef15f2e65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:32:32.334670 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.334650 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-model-cache\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:32:32.334670 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.334669 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-tmp-dir\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:32:32.334775 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.334677 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-home\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:32:32.336393 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.336367 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8975433-5603-4ad2-9840-492ef15f2e65-kube-api-access-gjtrs" (OuterVolumeSpecName: "kube-api-access-gjtrs") pod "e8975433-5603-4ad2-9840-492ef15f2e65" (UID: "e8975433-5603-4ad2-9840-492ef15f2e65"). InnerVolumeSpecName "kube-api-access-gjtrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:32:32.336512 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.336400 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8975433-5603-4ad2-9840-492ef15f2e65-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8975433-5603-4ad2-9840-492ef15f2e65" (UID: "e8975433-5603-4ad2-9840-492ef15f2e65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 14:32:32.336512 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.336429 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-dshm" (OuterVolumeSpecName: "dshm") pod "e8975433-5603-4ad2-9840-492ef15f2e65" (UID: "e8975433-5603-4ad2-9840-492ef15f2e65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:32:32.436067 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.436029 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8975433-5603-4ad2-9840-492ef15f2e65-tls-certs\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:32:32.436067 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.436059 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8975433-5603-4ad2-9840-492ef15f2e65-dshm\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:32:32.436067 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:32.436068 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjtrs\" (UniqueName: \"kubernetes.io/projected/e8975433-5603-4ad2-9840-492ef15f2e65-kube-api-access-gjtrs\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:32:33.048439 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:33.048408 2576 generic.go:356] "Generic (PLEG): container finished" podID="e8975433-5603-4ad2-9840-492ef15f2e65" containerID="305e81d29df0d2c98b22a424126631831ad99e9aeb35e2b00f32f74620300109" exitCode=0 Jul 10 14:32:33.048836 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:33.048459 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" Jul 10 14:32:33.048836 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:33.048484 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" event={"ID":"e8975433-5603-4ad2-9840-492ef15f2e65","Type":"ContainerDied","Data":"305e81d29df0d2c98b22a424126631831ad99e9aeb35e2b00f32f74620300109"} Jul 10 14:32:33.048836 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:33.048526 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl" event={"ID":"e8975433-5603-4ad2-9840-492ef15f2e65","Type":"ContainerDied","Data":"e934bc4c472a0fe5dd956886497cc3974ef340864b8b00841378399eb1be52be"} Jul 10 14:32:33.048836 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:33.048543 2576 scope.go:117] "RemoveContainer" containerID="305e81d29df0d2c98b22a424126631831ad99e9aeb35e2b00f32f74620300109" Jul 10 14:32:33.058217 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:33.058195 2576 scope.go:117] "RemoveContainer" containerID="305e81d29df0d2c98b22a424126631831ad99e9aeb35e2b00f32f74620300109" Jul 10 14:32:33.058484 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:32:33.058468 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"305e81d29df0d2c98b22a424126631831ad99e9aeb35e2b00f32f74620300109\": container with ID starting with 305e81d29df0d2c98b22a424126631831ad99e9aeb35e2b00f32f74620300109 not found: ID does not exist" containerID="305e81d29df0d2c98b22a424126631831ad99e9aeb35e2b00f32f74620300109" Jul 10 14:32:33.058536 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:33.058492 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305e81d29df0d2c98b22a424126631831ad99e9aeb35e2b00f32f74620300109"} err="failed to get container status \"305e81d29df0d2c98b22a424126631831ad99e9aeb35e2b00f32f74620300109\": rpc error: code = NotFound desc = could not find container \"305e81d29df0d2c98b22a424126631831ad99e9aeb35e2b00f32f74620300109\": container with ID starting with 305e81d29df0d2c98b22a424126631831ad99e9aeb35e2b00f32f74620300109 not found: ID does not exist" Jul 10 14:32:33.067647 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:33.067618 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl"] Jul 10 14:32:33.071561 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:33.071535 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-47e604/auth-test-model-47e604-kserve-5677bf6875-dnsjl"] Jul 10 14:32:35.036363 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:35.036324 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8975433-5603-4ad2-9840-492ef15f2e65" path="/var/lib/kubelet/pods/e8975433-5603-4ad2-9840-492ef15f2e65/volumes" Jul 10 14:32:35.305924 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:35.305830 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:32:46.713392 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.713356 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz"] Jul 10 14:32:46.713763 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.713738 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8975433-5603-4ad2-9840-492ef15f2e65" containerName="main" Jul 10 14:32:46.713763 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.713752 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8975433-5603-4ad2-9840-492ef15f2e65" containerName="main" Jul 10 14:32:46.713862 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.713839 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8975433-5603-4ad2-9840-492ef15f2e65" containerName="main" Jul 10 14:32:46.718033 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.718005 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:46.722074 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.722050 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-47e604\"/\"default-dockercfg-96rhq\"" Jul 10 14:32:46.722235 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.722096 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-47e604\"/\"sub-test-model-47e604-kserve-self-signed-certs\"" Jul 10 14:32:46.722235 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.722052 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-47e604\"/\"kube-root-ca.crt\"" Jul 10 14:32:46.722235 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.722052 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-47e604\"/\"openshift-service-ca.crt\"" Jul 10 14:32:46.727626 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.727604 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz"] Jul 10 14:32:46.764708 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.764672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h7sc\" (UniqueName: \"kubernetes.io/projected/db0edf1b-fc85-473a-bea2-89fb105d444e-kube-api-access-6h7sc\") pod \"sub-test-model-47e604-kserve-766d6d7755-vd7zz\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:46.764926 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.764715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-tmp-dir\") pod \"sub-test-model-47e604-kserve-766d6d7755-vd7zz\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:46.764926 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.764737 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-dshm\") pod \"sub-test-model-47e604-kserve-766d6d7755-vd7zz\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:46.764926 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.764837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-model-cache\") pod \"sub-test-model-47e604-kserve-766d6d7755-vd7zz\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:46.764926 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.764885 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-home\") pod \"sub-test-model-47e604-kserve-766d6d7755-vd7zz\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:46.764926 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.764909 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db0edf1b-fc85-473a-bea2-89fb105d444e-tls-certs\") pod \"sub-test-model-47e604-kserve-766d6d7755-vd7zz\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:46.768659 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.768629 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:32:46.866213 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.866182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h7sc\" (UniqueName: \"kubernetes.io/projected/db0edf1b-fc85-473a-bea2-89fb105d444e-kube-api-access-6h7sc\") pod \"sub-test-model-47e604-kserve-766d6d7755-vd7zz\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:46.866410 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.866222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-tmp-dir\") pod \"sub-test-model-47e604-kserve-766d6d7755-vd7zz\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:46.866410 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.866246 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-dshm\") pod \"sub-test-model-47e604-kserve-766d6d7755-vd7zz\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:46.866410 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.866266 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-model-cache\") pod \"sub-test-model-47e604-kserve-766d6d7755-vd7zz\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:46.866583 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.866403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-home\") pod \"sub-test-model-47e604-kserve-766d6d7755-vd7zz\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:46.866583 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.866454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db0edf1b-fc85-473a-bea2-89fb105d444e-tls-certs\") pod \"sub-test-model-47e604-kserve-766d6d7755-vd7zz\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:46.866692 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.866618 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-model-cache\") pod \"sub-test-model-47e604-kserve-766d6d7755-vd7zz\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:46.866692 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.866670 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-tmp-dir\") pod \"sub-test-model-47e604-kserve-766d6d7755-vd7zz\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:46.866803 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.866756 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-home\") pod \"sub-test-model-47e604-kserve-766d6d7755-vd7zz\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:46.868505 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.868483 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-dshm\") pod \"sub-test-model-47e604-kserve-766d6d7755-vd7zz\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:46.868901 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.868883 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db0edf1b-fc85-473a-bea2-89fb105d444e-tls-certs\") pod \"sub-test-model-47e604-kserve-766d6d7755-vd7zz\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:46.876930 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:46.876902 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h7sc\" (UniqueName: \"kubernetes.io/projected/db0edf1b-fc85-473a-bea2-89fb105d444e-kube-api-access-6h7sc\") pod \"sub-test-model-47e604-kserve-766d6d7755-vd7zz\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:47.027891 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:47.027774 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:47.173419 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:47.173392 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz"] Jul 10 14:32:47.175631 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:32:47.175607 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb0edf1b_fc85_473a_bea2_89fb105d444e.slice/crio-e4f30c2d9dda4dcff187748ee0468210db3242fb870d3ae8e8ceb9dc54678237 WatchSource:0}: Error finding container e4f30c2d9dda4dcff187748ee0468210db3242fb870d3ae8e8ceb9dc54678237: Status 404 returned error can't find the container with id e4f30c2d9dda4dcff187748ee0468210db3242fb870d3ae8e8ceb9dc54678237 Jul 10 14:32:48.124972 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:48.124937 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" event={"ID":"db0edf1b-fc85-473a-bea2-89fb105d444e","Type":"ContainerStarted","Data":"6a08ed60943db324c7a5fc49ef2abec2967feb024f264316af6ddbdc8739c1b6"} Jul 10 14:32:48.125381 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:48.124979 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" event={"ID":"db0edf1b-fc85-473a-bea2-89fb105d444e","Type":"ContainerStarted","Data":"e4f30c2d9dda4dcff187748ee0468210db3242fb870d3ae8e8ceb9dc54678237"} Jul 10 14:32:48.144360 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:48.144308 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" podStartSLOduration=2.144292922 podStartE2EDuration="2.144292922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:32:48.143426197 +0000 UTC m=+2575.607031644" watchObservedRunningTime="2026-07-10 14:32:48.144292922 +0000 UTC m=+2575.607898368" Jul 10 14:32:51.401830 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:51.401784 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:32:57.028630 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:57.028591 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:57.029401 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:57.028898 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:57.042223 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:57.042193 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:32:57.180526 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:32:57.180492 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:33:06.311756 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:06.311720 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:33:26.013089 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:26.012996 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:33:32.326000 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:32.325961 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:33:33.127013 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:33.126981 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz"] Jul 10 14:33:33.193685 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:33.193652 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:33:39.000321 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:39.000281 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:33:45.518851 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:45.518785 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:33:48.146820 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.146735 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" podUID="db0edf1b-fc85-473a-bea2-89fb105d444e" containerName="main" containerID="cri-o://6a08ed60943db324c7a5fc49ef2abec2967feb024f264316af6ddbdc8739c1b6" gracePeriod=45 Jul 10 14:33:48.398636 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.398562 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:33:48.422265 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.422216 2576 generic.go:356] "Generic (PLEG): container finished" podID="db0edf1b-fc85-473a-bea2-89fb105d444e" containerID="6a08ed60943db324c7a5fc49ef2abec2967feb024f264316af6ddbdc8739c1b6" exitCode=0 Jul 10 14:33:48.422265 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.422267 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" Jul 10 14:33:48.422498 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.422282 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" event={"ID":"db0edf1b-fc85-473a-bea2-89fb105d444e","Type":"ContainerDied","Data":"6a08ed60943db324c7a5fc49ef2abec2967feb024f264316af6ddbdc8739c1b6"} Jul 10 14:33:48.422498 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.422331 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz" event={"ID":"db0edf1b-fc85-473a-bea2-89fb105d444e","Type":"ContainerDied","Data":"e4f30c2d9dda4dcff187748ee0468210db3242fb870d3ae8e8ceb9dc54678237"} Jul 10 14:33:48.422498 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.422354 2576 scope.go:117] "RemoveContainer" containerID="6a08ed60943db324c7a5fc49ef2abec2967feb024f264316af6ddbdc8739c1b6" Jul 10 14:33:48.432880 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.432856 2576 scope.go:117] "RemoveContainer" containerID="6a08ed60943db324c7a5fc49ef2abec2967feb024f264316af6ddbdc8739c1b6" Jul 10 14:33:48.433221 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:33:48.433192 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a08ed60943db324c7a5fc49ef2abec2967feb024f264316af6ddbdc8739c1b6\": container with ID starting with 6a08ed60943db324c7a5fc49ef2abec2967feb024f264316af6ddbdc8739c1b6 not found: ID does not exist" containerID="6a08ed60943db324c7a5fc49ef2abec2967feb024f264316af6ddbdc8739c1b6" Jul 10 14:33:48.433280 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.433231 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a08ed60943db324c7a5fc49ef2abec2967feb024f264316af6ddbdc8739c1b6"} err="failed to get container status \"6a08ed60943db324c7a5fc49ef2abec2967feb024f264316af6ddbdc8739c1b6\": rpc error: code = NotFound desc = could not find container \"6a08ed60943db324c7a5fc49ef2abec2967feb024f264316af6ddbdc8739c1b6\": container with ID starting with 6a08ed60943db324c7a5fc49ef2abec2967feb024f264316af6ddbdc8739c1b6 not found: ID does not exist" Jul 10 14:33:48.546017 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.545987 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h7sc\" (UniqueName: \"kubernetes.io/projected/db0edf1b-fc85-473a-bea2-89fb105d444e-kube-api-access-6h7sc\") pod \"db0edf1b-fc85-473a-bea2-89fb105d444e\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " Jul 10 14:33:48.546207 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.546050 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-dshm\") pod \"db0edf1b-fc85-473a-bea2-89fb105d444e\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " Jul 10 14:33:48.546207 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.546065 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-home\") pod \"db0edf1b-fc85-473a-bea2-89fb105d444e\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " Jul 10 14:33:48.546207 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.546100 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-model-cache\") pod \"db0edf1b-fc85-473a-bea2-89fb105d444e\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " Jul 10 14:33:48.546207 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.546185 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db0edf1b-fc85-473a-bea2-89fb105d444e-tls-certs\") pod \"db0edf1b-fc85-473a-bea2-89fb105d444e\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " Jul 10 14:33:48.546421 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.546220 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-tmp-dir\") pod \"db0edf1b-fc85-473a-bea2-89fb105d444e\" (UID: \"db0edf1b-fc85-473a-bea2-89fb105d444e\") " Jul 10 14:33:48.546421 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.546376 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-home" (OuterVolumeSpecName: "home") pod "db0edf1b-fc85-473a-bea2-89fb105d444e" (UID: "db0edf1b-fc85-473a-bea2-89fb105d444e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:33:48.546531 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.546415 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-model-cache" (OuterVolumeSpecName: "model-cache") pod "db0edf1b-fc85-473a-bea2-89fb105d444e" (UID: "db0edf1b-fc85-473a-bea2-89fb105d444e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:33:48.546595 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.546580 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-home\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:33:48.546663 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.546597 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-model-cache\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:33:48.546663 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.546599 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db0edf1b-fc85-473a-bea2-89fb105d444e" (UID: "db0edf1b-fc85-473a-bea2-89fb105d444e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:33:48.548232 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.548207 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0edf1b-fc85-473a-bea2-89fb105d444e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db0edf1b-fc85-473a-bea2-89fb105d444e" (UID: "db0edf1b-fc85-473a-bea2-89fb105d444e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 14:33:48.548381 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.548240 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-dshm" (OuterVolumeSpecName: "dshm") pod "db0edf1b-fc85-473a-bea2-89fb105d444e" (UID: "db0edf1b-fc85-473a-bea2-89fb105d444e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:33:48.548381 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.548339 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0edf1b-fc85-473a-bea2-89fb105d444e-kube-api-access-6h7sc" (OuterVolumeSpecName: "kube-api-access-6h7sc") pod "db0edf1b-fc85-473a-bea2-89fb105d444e" (UID: "db0edf1b-fc85-473a-bea2-89fb105d444e"). InnerVolumeSpecName "kube-api-access-6h7sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:33:48.648055 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.648015 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-dshm\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:33:48.648055 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.648048 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db0edf1b-fc85-473a-bea2-89fb105d444e-tls-certs\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:33:48.648055 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.648062 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db0edf1b-fc85-473a-bea2-89fb105d444e-tmp-dir\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:33:48.648315 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.648074 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h7sc\" (UniqueName: \"kubernetes.io/projected/db0edf1b-fc85-473a-bea2-89fb105d444e-kube-api-access-6h7sc\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:33:48.747104 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.747054 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz"] Jul 10 14:33:48.749567 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:48.749546 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-47e604/sub-test-model-47e604-kserve-766d6d7755-vd7zz"] Jul 10 14:33:49.038215 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:33:49.038105 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db0edf1b-fc85-473a-bea2-89fb105d444e" path="/var/lib/kubelet/pods/db0edf1b-fc85-473a-bea2-89fb105d444e/volumes" Jul 10 14:34:00.969659 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:00.969620 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:34:05.111508 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:05.111469 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:34:18.846037 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:18.845994 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:34:18.947758 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:18.947726 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:34:19.616114 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:19.616075 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:34:37.906631 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:37.906584 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:34:45.217346 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:45.217263 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:34:46.871070 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:46.871035 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:34:46.937193 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:46.937157 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7289-140576860919344"] Jul 10 14:34:46.937527 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:46.937514 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db0edf1b-fc85-473a-bea2-89fb105d444e" containerName="main" Jul 10 14:34:46.937581 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:46.937529 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="db0edf1b-fc85-473a-bea2-89fb105d444e" containerName="main" Jul 10 14:34:46.937618 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:46.937603 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="db0edf1b-fc85-473a-bea2-89fb105d444e" containerName="main" Jul 10 14:34:46.940751 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:46.940732 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576860919344" Jul 10 14:34:46.943453 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:46.943430 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-5dfxb\"" Jul 10 14:34:46.946434 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:46.946411 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7289-140576860919344"] Jul 10 14:34:46.978567 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:46.978524 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdv85\" (UniqueName: \"kubernetes.io/projected/b91360a5-594e-4b0f-b0eb-74d746c75040-kube-api-access-kdv85\") pod \"test-curl-7289-140576860919344\" (UID: \"b91360a5-594e-4b0f-b0eb-74d746c75040\") " pod="openshift-ingress/test-curl-7289-140576860919344" Jul 10 14:34:47.079470 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:47.079435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdv85\" (UniqueName: \"kubernetes.io/projected/b91360a5-594e-4b0f-b0eb-74d746c75040-kube-api-access-kdv85\") pod \"test-curl-7289-140576860919344\" (UID: \"b91360a5-594e-4b0f-b0eb-74d746c75040\") " pod="openshift-ingress/test-curl-7289-140576860919344" Jul 10 14:34:47.088725 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:47.088695 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdv85\" (UniqueName: \"kubernetes.io/projected/b91360a5-594e-4b0f-b0eb-74d746c75040-kube-api-access-kdv85\") pod \"test-curl-7289-140576860919344\" (UID: \"b91360a5-594e-4b0f-b0eb-74d746c75040\") " pod="openshift-ingress/test-curl-7289-140576860919344" Jul 10 14:34:47.250426 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:47.250379 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576860919344" Jul 10 14:34:47.388130 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:47.388102 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7289-140576860919344"] Jul 10 14:34:47.390438 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:34:47.390410 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb91360a5_594e_4b0f_b0eb_74d746c75040.slice/crio-9d3c43c17490f14b722e03d33e6504dcf75ec11a7f07fa066fa1d92a6acf2b51 WatchSource:0}: Error finding container 9d3c43c17490f14b722e03d33e6504dcf75ec11a7f07fa066fa1d92a6acf2b51: Status 404 returned error can't find the container with id 9d3c43c17490f14b722e03d33e6504dcf75ec11a7f07fa066fa1d92a6acf2b51 Jul 10 14:34:47.391630 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:47.391614 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 14:34:47.754631 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:47.754592 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7289-140576860919344" event={"ID":"b91360a5-594e-4b0f-b0eb-74d746c75040","Type":"ContainerStarted","Data":"9d3c43c17490f14b722e03d33e6504dcf75ec11a7f07fa066fa1d92a6acf2b51"} Jul 10 14:34:48.760868 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:48.760796 2576 generic.go:356] "Generic (PLEG): container finished" podID="b91360a5-594e-4b0f-b0eb-74d746c75040" containerID="962419cddc0b6d7fb93587b27ecde0933787d784f4bd46dd75f73ff0e2c968ce" exitCode=0 Jul 10 14:34:48.761271 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:48.760888 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7289-140576860919344" event={"ID":"b91360a5-594e-4b0f-b0eb-74d746c75040","Type":"ContainerDied","Data":"962419cddc0b6d7fb93587b27ecde0933787d784f4bd46dd75f73ff0e2c968ce"} Jul 10 14:34:49.903333 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:49.903307 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576860919344" Jul 10 14:34:49.925247 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:49.925218 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7289-140576860919344_b91360a5-594e-4b0f-b0eb-74d746c75040/test-curl-7289-140576860919344/0.log" Jul 10 14:34:49.948907 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:49.948872 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7289-140576860919344"] Jul 10 14:34:49.953085 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:49.953061 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7289-140576860919344"] Jul 10 14:34:50.006540 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:50.006508 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdv85\" (UniqueName: \"kubernetes.io/projected/b91360a5-594e-4b0f-b0eb-74d746c75040-kube-api-access-kdv85\") pod \"b91360a5-594e-4b0f-b0eb-74d746c75040\" (UID: \"b91360a5-594e-4b0f-b0eb-74d746c75040\") " Jul 10 14:34:50.008657 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:50.008632 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91360a5-594e-4b0f-b0eb-74d746c75040-kube-api-access-kdv85" (OuterVolumeSpecName: "kube-api-access-kdv85") pod "b91360a5-594e-4b0f-b0eb-74d746c75040" (UID: "b91360a5-594e-4b0f-b0eb-74d746c75040"). InnerVolumeSpecName "kube-api-access-kdv85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:34:50.085761 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:50.085683 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7289-140576861036592"] Jul 10 14:34:50.086038 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:50.086025 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b91360a5-594e-4b0f-b0eb-74d746c75040" containerName="test-curl-7289-140576860919344" Jul 10 14:34:50.086133 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:50.086041 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b91360a5-594e-4b0f-b0eb-74d746c75040" containerName="test-curl-7289-140576860919344" Jul 10 14:34:50.086133 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:50.086103 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b91360a5-594e-4b0f-b0eb-74d746c75040" containerName="test-curl-7289-140576860919344" Jul 10 14:34:50.089135 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:50.089118 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576861036592" Jul 10 14:34:50.095422 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:50.095393 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7289-140576861036592"] Jul 10 14:34:50.107546 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:50.107518 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdv85\" (UniqueName: \"kubernetes.io/projected/b91360a5-594e-4b0f-b0eb-74d746c75040-kube-api-access-kdv85\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:34:50.208126 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:50.207996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrnp9\" (UniqueName: \"kubernetes.io/projected/902d9d77-bb64-46db-b5ea-1f9a4bf4189d-kube-api-access-qrnp9\") pod \"test-curl-7289-140576861036592\" (UID: \"902d9d77-bb64-46db-b5ea-1f9a4bf4189d\") " pod="openshift-ingress/test-curl-7289-140576861036592" Jul 10 14:34:50.309068 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:50.309028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrnp9\" (UniqueName: \"kubernetes.io/projected/902d9d77-bb64-46db-b5ea-1f9a4bf4189d-kube-api-access-qrnp9\") pod \"test-curl-7289-140576861036592\" (UID: \"902d9d77-bb64-46db-b5ea-1f9a4bf4189d\") " pod="openshift-ingress/test-curl-7289-140576861036592" Jul 10 14:34:50.318073 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:50.318047 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrnp9\" (UniqueName: \"kubernetes.io/projected/902d9d77-bb64-46db-b5ea-1f9a4bf4189d-kube-api-access-qrnp9\") pod \"test-curl-7289-140576861036592\" (UID: \"902d9d77-bb64-46db-b5ea-1f9a4bf4189d\") " pod="openshift-ingress/test-curl-7289-140576861036592" Jul 10 14:34:50.399268 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:50.399229 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576861036592" Jul 10 14:34:50.533049 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:50.532944 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7289-140576861036592"] Jul 10 14:34:50.535115 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:34:50.535085 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod902d9d77_bb64_46db_b5ea_1f9a4bf4189d.slice/crio-e9f9c97199f1e71ed5d5e3735df4c4c1ffc6e5e1ed50768d84db6e442f9fe627 WatchSource:0}: Error finding container e9f9c97199f1e71ed5d5e3735df4c4c1ffc6e5e1ed50768d84db6e442f9fe627: Status 404 returned error can't find the container with id e9f9c97199f1e71ed5d5e3735df4c4c1ffc6e5e1ed50768d84db6e442f9fe627 Jul 10 14:34:50.772793 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:50.772686 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7289-140576861036592" event={"ID":"902d9d77-bb64-46db-b5ea-1f9a4bf4189d","Type":"ContainerStarted","Data":"e9f9c97199f1e71ed5d5e3735df4c4c1ffc6e5e1ed50768d84db6e442f9fe627"} Jul 10 14:34:50.773916 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:50.773897 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d3c43c17490f14b722e03d33e6504dcf75ec11a7f07fa066fa1d92a6acf2b51" Jul 10 14:34:50.774023 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:50.773933 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576860919344" Jul 10 14:34:50.784733 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:34:50.784705 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7289-140576860919344\" is forbidden: User \"system:node:ip-10-0-143-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-143-25.ec2.internal' and this object" podUID="b91360a5-594e-4b0f-b0eb-74d746c75040" pod="openshift-ingress/test-curl-7289-140576860919344" Jul 10 14:34:51.037067 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:51.036984 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b91360a5-594e-4b0f-b0eb-74d746c75040" path="/var/lib/kubelet/pods/b91360a5-594e-4b0f-b0eb-74d746c75040/volumes" Jul 10 14:34:51.779678 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:51.779647 2576 generic.go:356] "Generic (PLEG): container finished" podID="902d9d77-bb64-46db-b5ea-1f9a4bf4189d" containerID="cbb347a45dfeb4ef099dd2190754797e149978fdcb3688bfe35c5659603cb4e9" exitCode=0 Jul 10 14:34:51.779885 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:51.779728 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7289-140576861036592" event={"ID":"902d9d77-bb64-46db-b5ea-1f9a4bf4189d","Type":"ContainerDied","Data":"cbb347a45dfeb4ef099dd2190754797e149978fdcb3688bfe35c5659603cb4e9"} Jul 10 14:34:51.907216 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:51.907179 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:34:52.930543 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:52.930514 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576861036592" Jul 10 14:34:52.952724 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:52.952697 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7289-140576861036592_902d9d77-bb64-46db-b5ea-1f9a4bf4189d/test-curl-7289-140576861036592/0.log" Jul 10 14:34:52.978773 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:52.978742 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7289-140576861036592"] Jul 10 14:34:52.982274 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:52.982246 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7289-140576861036592"] Jul 10 14:34:53.034785 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.034758 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrnp9\" (UniqueName: \"kubernetes.io/projected/902d9d77-bb64-46db-b5ea-1f9a4bf4189d-kube-api-access-qrnp9\") pod \"902d9d77-bb64-46db-b5ea-1f9a4bf4189d\" (UID: \"902d9d77-bb64-46db-b5ea-1f9a4bf4189d\") " Jul 10 14:34:53.036759 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.036689 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902d9d77-bb64-46db-b5ea-1f9a4bf4189d-kube-api-access-qrnp9" (OuterVolumeSpecName: "kube-api-access-qrnp9") pod "902d9d77-bb64-46db-b5ea-1f9a4bf4189d" (UID: "902d9d77-bb64-46db-b5ea-1f9a4bf4189d"). InnerVolumeSpecName "kube-api-access-qrnp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:34:53.135421 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.135382 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrnp9\" (UniqueName: \"kubernetes.io/projected/902d9d77-bb64-46db-b5ea-1f9a4bf4189d-kube-api-access-qrnp9\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:34:53.142047 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.142012 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7289-140576863132528"] Jul 10 14:34:53.142357 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.142345 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902d9d77-bb64-46db-b5ea-1f9a4bf4189d" containerName="test-curl-7289-140576861036592" Jul 10 14:34:53.142404 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.142359 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="902d9d77-bb64-46db-b5ea-1f9a4bf4189d" containerName="test-curl-7289-140576861036592" Jul 10 14:34:53.142440 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.142416 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="902d9d77-bb64-46db-b5ea-1f9a4bf4189d" containerName="test-curl-7289-140576861036592" Jul 10 14:34:53.145302 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.145285 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576863132528" Jul 10 14:34:53.152356 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.152316 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7289-140576863132528"] Jul 10 14:34:53.190364 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.190326 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 14:34:53.197277 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.197253 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 14:34:53.236689 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.236651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnw6c\" (UniqueName: \"kubernetes.io/projected/2283293f-df13-42ef-85e8-9db5afc2630e-kube-api-access-rnw6c\") pod \"test-curl-7289-140576863132528\" (UID: \"2283293f-df13-42ef-85e8-9db5afc2630e\") " pod="openshift-ingress/test-curl-7289-140576863132528" Jul 10 14:34:53.337236 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.337145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnw6c\" (UniqueName: \"kubernetes.io/projected/2283293f-df13-42ef-85e8-9db5afc2630e-kube-api-access-rnw6c\") pod \"test-curl-7289-140576863132528\" (UID: \"2283293f-df13-42ef-85e8-9db5afc2630e\") " pod="openshift-ingress/test-curl-7289-140576863132528" Jul 10 14:34:53.345502 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.345462 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnw6c\" (UniqueName: \"kubernetes.io/projected/2283293f-df13-42ef-85e8-9db5afc2630e-kube-api-access-rnw6c\") pod \"test-curl-7289-140576863132528\" (UID: \"2283293f-df13-42ef-85e8-9db5afc2630e\") " pod="openshift-ingress/test-curl-7289-140576863132528" Jul 10 14:34:53.455797 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.455747 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576863132528" Jul 10 14:34:53.589493 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.589446 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7289-140576863132528"] Jul 10 14:34:53.591625 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:34:53.591589 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2283293f_df13_42ef_85e8_9db5afc2630e.slice/crio-e3702c28bbbd039fe5f2dc9e170ce3f102fb8cae9059b3bb225268239ae75275 WatchSource:0}: Error finding container e3702c28bbbd039fe5f2dc9e170ce3f102fb8cae9059b3bb225268239ae75275: Status 404 returned error can't find the container with id e3702c28bbbd039fe5f2dc9e170ce3f102fb8cae9059b3bb225268239ae75275 Jul 10 14:34:53.791036 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.790998 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7289-140576863132528" event={"ID":"2283293f-df13-42ef-85e8-9db5afc2630e","Type":"ContainerStarted","Data":"e3702c28bbbd039fe5f2dc9e170ce3f102fb8cae9059b3bb225268239ae75275"} Jul 10 14:34:53.792108 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.792090 2576 scope.go:117] "RemoveContainer" containerID="cbb347a45dfeb4ef099dd2190754797e149978fdcb3688bfe35c5659603cb4e9" Jul 10 14:34:53.792219 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:53.792136 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576861036592" Jul 10 14:34:54.190403 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:54.190367 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:34:54.311595 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:54.311557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:34:54.799856 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:54.799823 2576 generic.go:356] "Generic (PLEG): container finished" podID="2283293f-df13-42ef-85e8-9db5afc2630e" containerID="36ae037cfde71c418cb77a8b8c6ae59fea9721f60020399d16bf35627d7e98c7" exitCode=0 Jul 10 14:34:54.800015 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:54.799908 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7289-140576863132528" event={"ID":"2283293f-df13-42ef-85e8-9db5afc2630e","Type":"ContainerDied","Data":"36ae037cfde71c418cb77a8b8c6ae59fea9721f60020399d16bf35627d7e98c7"} Jul 10 14:34:55.036391 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:55.036359 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="902d9d77-bb64-46db-b5ea-1f9a4bf4189d" path="/var/lib/kubelet/pods/902d9d77-bb64-46db-b5ea-1f9a4bf4189d/volumes" Jul 10 14:34:55.945258 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:55.945231 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576863132528" Jul 10 14:34:55.967558 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:55.967535 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7289-140576863132528_2283293f-df13-42ef-85e8-9db5afc2630e/test-curl-7289-140576863132528/0.log" Jul 10 14:34:55.995553 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:55.995519 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7289-140576863132528"] Jul 10 14:34:55.999655 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:55.999630 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7289-140576863132528"] Jul 10 14:34:56.062381 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.062279 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnw6c\" (UniqueName: \"kubernetes.io/projected/2283293f-df13-42ef-85e8-9db5afc2630e-kube-api-access-rnw6c\") pod \"2283293f-df13-42ef-85e8-9db5afc2630e\" (UID: \"2283293f-df13-42ef-85e8-9db5afc2630e\") " Jul 10 14:34:56.064366 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.064336 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2283293f-df13-42ef-85e8-9db5afc2630e-kube-api-access-rnw6c" (OuterVolumeSpecName: "kube-api-access-rnw6c") pod "2283293f-df13-42ef-85e8-9db5afc2630e" (UID: "2283293f-df13-42ef-85e8-9db5afc2630e"). InnerVolumeSpecName "kube-api-access-rnw6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:34:56.115904 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.115868 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7289-140576863132528"] Jul 10 14:34:56.116212 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.116200 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2283293f-df13-42ef-85e8-9db5afc2630e" containerName="test-curl-7289-140576863132528" Jul 10 14:34:56.116276 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.116214 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2283293f-df13-42ef-85e8-9db5afc2630e" containerName="test-curl-7289-140576863132528" Jul 10 14:34:56.116310 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.116293 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2283293f-df13-42ef-85e8-9db5afc2630e" containerName="test-curl-7289-140576863132528" Jul 10 14:34:56.119226 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.119208 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576863132528" Jul 10 14:34:56.127156 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.127129 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7289-140576863132528"] Jul 10 14:34:56.163498 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.163457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfnf2\" (UniqueName: \"kubernetes.io/projected/a06e300e-2486-4ba2-9482-9dbbc9e8cd3f-kube-api-access-lfnf2\") pod \"test-curl-7289-140576863132528\" (UID: \"a06e300e-2486-4ba2-9482-9dbbc9e8cd3f\") " pod="openshift-ingress/test-curl-7289-140576863132528" Jul 10 14:34:56.163650 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.163616 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnw6c\" (UniqueName: \"kubernetes.io/projected/2283293f-df13-42ef-85e8-9db5afc2630e-kube-api-access-rnw6c\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:34:56.264186 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.264144 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfnf2\" (UniqueName: \"kubernetes.io/projected/a06e300e-2486-4ba2-9482-9dbbc9e8cd3f-kube-api-access-lfnf2\") pod \"test-curl-7289-140576863132528\" (UID: \"a06e300e-2486-4ba2-9482-9dbbc9e8cd3f\") " pod="openshift-ingress/test-curl-7289-140576863132528" Jul 10 14:34:56.273589 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.273556 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfnf2\" (UniqueName: \"kubernetes.io/projected/a06e300e-2486-4ba2-9482-9dbbc9e8cd3f-kube-api-access-lfnf2\") pod \"test-curl-7289-140576863132528\" (UID: \"a06e300e-2486-4ba2-9482-9dbbc9e8cd3f\") " pod="openshift-ingress/test-curl-7289-140576863132528" Jul 10 14:34:56.429568 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.429534 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576863132528" Jul 10 14:34:56.564708 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.564683 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7289-140576863132528"] Jul 10 14:34:56.566724 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:34:56.566692 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda06e300e_2486_4ba2_9482_9dbbc9e8cd3f.slice/crio-599868174fdf688bf6a47138002fa78086ad2b0738ea46eebecb04c1ce2c9385 WatchSource:0}: Error finding container 599868174fdf688bf6a47138002fa78086ad2b0738ea46eebecb04c1ce2c9385: Status 404 returned error can't find the container with id 599868174fdf688bf6a47138002fa78086ad2b0738ea46eebecb04c1ce2c9385 Jul 10 14:34:56.598902 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.598873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:34:56.810681 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.810582 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576863132528" Jul 10 14:34:56.810681 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.810628 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3702c28bbbd039fe5f2dc9e170ce3f102fb8cae9059b3bb225268239ae75275" Jul 10 14:34:56.811762 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.811729 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7289-140576863132528" event={"ID":"a06e300e-2486-4ba2-9482-9dbbc9e8cd3f","Type":"ContainerStarted","Data":"599868174fdf688bf6a47138002fa78086ad2b0738ea46eebecb04c1ce2c9385"} Jul 10 14:34:56.828969 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:56.828927 2576 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7289-140576863132528" oldPodUID="2283293f-df13-42ef-85e8-9db5afc2630e" podUID="a06e300e-2486-4ba2-9482-9dbbc9e8cd3f" Jul 10 14:34:57.037018 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:57.036987 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2283293f-df13-42ef-85e8-9db5afc2630e" path="/var/lib/kubelet/pods/2283293f-df13-42ef-85e8-9db5afc2630e/volumes" Jul 10 14:34:57.817775 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:57.817739 2576 generic.go:356] "Generic (PLEG): container finished" podID="a06e300e-2486-4ba2-9482-9dbbc9e8cd3f" containerID="4284d0cba8d6735cf04acb80097330db9ee6c49d05c7a3b66e3a2576f9273991" exitCode=0 Jul 10 14:34:57.817986 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:57.817837 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7289-140576863132528" event={"ID":"a06e300e-2486-4ba2-9482-9dbbc9e8cd3f","Type":"ContainerDied","Data":"4284d0cba8d6735cf04acb80097330db9ee6c49d05c7a3b66e3a2576f9273991"} Jul 10 14:34:58.971862 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:58.971839 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576863132528" Jul 10 14:34:58.994602 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:58.994563 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7289-140576863132528_a06e300e-2486-4ba2-9482-9dbbc9e8cd3f/test-curl-7289-140576863132528/0.log" Jul 10 14:34:59.022318 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.022283 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7289-140576863132528"] Jul 10 14:34:59.026356 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.026328 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7289-140576863132528"] Jul 10 14:34:59.089381 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.089275 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfnf2\" (UniqueName: \"kubernetes.io/projected/a06e300e-2486-4ba2-9482-9dbbc9e8cd3f-kube-api-access-lfnf2\") pod \"a06e300e-2486-4ba2-9482-9dbbc9e8cd3f\" (UID: \"a06e300e-2486-4ba2-9482-9dbbc9e8cd3f\") " Jul 10 14:34:59.091332 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.091306 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06e300e-2486-4ba2-9482-9dbbc9e8cd3f-kube-api-access-lfnf2" (OuterVolumeSpecName: "kube-api-access-lfnf2") pod "a06e300e-2486-4ba2-9482-9dbbc9e8cd3f" (UID: "a06e300e-2486-4ba2-9482-9dbbc9e8cd3f"). InnerVolumeSpecName "kube-api-access-lfnf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:34:59.146273 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.146239 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7289-140576863133536"] Jul 10 14:34:59.146659 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.146643 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a06e300e-2486-4ba2-9482-9dbbc9e8cd3f" containerName="test-curl-7289-140576863132528" Jul 10 14:34:59.146708 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.146663 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a06e300e-2486-4ba2-9482-9dbbc9e8cd3f" containerName="test-curl-7289-140576863132528" Jul 10 14:34:59.146743 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.146726 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a06e300e-2486-4ba2-9482-9dbbc9e8cd3f" containerName="test-curl-7289-140576863132528" Jul 10 14:34:59.149953 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.149937 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576863133536" Jul 10 14:34:59.158131 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.158107 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7289-140576863133536"] Jul 10 14:34:59.190481 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.190447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rnfv\" (UniqueName: \"kubernetes.io/projected/d2fdf737-7866-4da4-82ee-cb521b2d2ceb-kube-api-access-6rnfv\") pod \"test-curl-7289-140576863133536\" (UID: \"d2fdf737-7866-4da4-82ee-cb521b2d2ceb\") " pod="openshift-ingress/test-curl-7289-140576863133536" Jul 10 14:34:59.190662 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.190583 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfnf2\" (UniqueName: \"kubernetes.io/projected/a06e300e-2486-4ba2-9482-9dbbc9e8cd3f-kube-api-access-lfnf2\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:34:59.291099 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.291057 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rnfv\" (UniqueName: \"kubernetes.io/projected/d2fdf737-7866-4da4-82ee-cb521b2d2ceb-kube-api-access-6rnfv\") pod \"test-curl-7289-140576863133536\" (UID: \"d2fdf737-7866-4da4-82ee-cb521b2d2ceb\") " pod="openshift-ingress/test-curl-7289-140576863133536" Jul 10 14:34:59.300197 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.300163 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rnfv\" (UniqueName: \"kubernetes.io/projected/d2fdf737-7866-4da4-82ee-cb521b2d2ceb-kube-api-access-6rnfv\") pod \"test-curl-7289-140576863133536\" (UID: \"d2fdf737-7866-4da4-82ee-cb521b2d2ceb\") " pod="openshift-ingress/test-curl-7289-140576863133536" Jul 10 14:34:59.459724 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.459687 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576863133536" Jul 10 14:34:59.611158 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.611129 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7289-140576863133536"] Jul 10 14:34:59.613086 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:34:59.613054 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2fdf737_7866_4da4_82ee_cb521b2d2ceb.slice/crio-c07e0b3572e695f5f7e0a45bc3c2c2e2af52add730446d112f2edb715440c613 WatchSource:0}: Error finding container c07e0b3572e695f5f7e0a45bc3c2c2e2af52add730446d112f2edb715440c613: Status 404 returned error can't find the container with id c07e0b3572e695f5f7e0a45bc3c2c2e2af52add730446d112f2edb715440c613 Jul 10 14:34:59.829858 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.829754 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576863132528" Jul 10 14:34:59.829858 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.829820 2576 scope.go:117] "RemoveContainer" containerID="4284d0cba8d6735cf04acb80097330db9ee6c49d05c7a3b66e3a2576f9273991" Jul 10 14:34:59.831144 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:34:59.831109 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7289-140576863133536" event={"ID":"d2fdf737-7866-4da4-82ee-cb521b2d2ceb","Type":"ContainerStarted","Data":"c07e0b3572e695f5f7e0a45bc3c2c2e2af52add730446d112f2edb715440c613"} Jul 10 14:35:00.837361 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:00.837325 2576 generic.go:356] "Generic (PLEG): container finished" podID="d2fdf737-7866-4da4-82ee-cb521b2d2ceb" containerID="a79a018ddd283373d293a109b5cccaea06df5789ed6a105f053a03acae6857d8" exitCode=0 Jul 10 14:35:00.837845 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:00.837409 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7289-140576863133536" event={"ID":"d2fdf737-7866-4da4-82ee-cb521b2d2ceb","Type":"ContainerDied","Data":"a79a018ddd283373d293a109b5cccaea06df5789ed6a105f053a03acae6857d8"} Jul 10 14:35:01.036046 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:01.036015 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a06e300e-2486-4ba2-9482-9dbbc9e8cd3f" path="/var/lib/kubelet/pods/a06e300e-2486-4ba2-9482-9dbbc9e8cd3f/volumes" Jul 10 14:35:01.987440 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:01.987413 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576863133536" Jul 10 14:35:02.012277 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:02.012247 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7289-140576863133536_d2fdf737-7866-4da4-82ee-cb521b2d2ceb/test-curl-7289-140576863133536/0.log" Jul 10 14:35:02.038747 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:02.038716 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7289-140576863133536"] Jul 10 14:35:02.047908 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:02.047871 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7289-140576863133536"] Jul 10 14:35:02.106023 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:02.105916 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:02.117290 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:02.117259 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rnfv\" (UniqueName: \"kubernetes.io/projected/d2fdf737-7866-4da4-82ee-cb521b2d2ceb-kube-api-access-6rnfv\") pod \"d2fdf737-7866-4da4-82ee-cb521b2d2ceb\" (UID: \"d2fdf737-7866-4da4-82ee-cb521b2d2ceb\") " Jul 10 14:35:02.119396 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:02.119368 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2fdf737-7866-4da4-82ee-cb521b2d2ceb-kube-api-access-6rnfv" (OuterVolumeSpecName: "kube-api-access-6rnfv") pod "d2fdf737-7866-4da4-82ee-cb521b2d2ceb" (UID: "d2fdf737-7866-4da4-82ee-cb521b2d2ceb"). InnerVolumeSpecName "kube-api-access-6rnfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:35:02.195048 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:02.195013 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:02.218673 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:02.218639 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rnfv\" (UniqueName: \"kubernetes.io/projected/d2fdf737-7866-4da4-82ee-cb521b2d2ceb-kube-api-access-6rnfv\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:35:02.851096 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:02.851061 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c07e0b3572e695f5f7e0a45bc3c2c2e2af52add730446d112f2edb715440c613" Jul 10 14:35:02.851096 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:02.851076 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7289-140576863133536" Jul 10 14:35:02.861069 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:35:02.861041 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7289-140576863133536\" is forbidden: User \"system:node:ip-10-0-143-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-143-25.ec2.internal' and this object" podUID="d2fdf737-7866-4da4-82ee-cb521b2d2ceb" pod="openshift-ingress/test-curl-7289-140576863133536" Jul 10 14:35:03.036678 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:35:03.036640 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7289-140576863133536\" is forbidden: User \"system:node:ip-10-0-143-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-143-25.ec2.internal' and this object" podUID="d2fdf737-7866-4da4-82ee-cb521b2d2ceb" pod="openshift-ingress/test-curl-7289-140576863133536" Jul 10 14:35:03.037762 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:03.037735 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2fdf737-7866-4da4-82ee-cb521b2d2ceb" path="/var/lib/kubelet/pods/d2fdf737-7866-4da4-82ee-cb521b2d2ceb/volumes" Jul 10 14:35:05.686131 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:05.686093 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:05.784043 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:05.784014 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:08.464139 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:08.464109 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:08.565548 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:08.565515 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:15.641595 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:15.641544 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:19.101750 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:19.101714 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:35:20.264010 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:20.263971 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:27.114661 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:27.114599 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:27.213548 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:27.213517 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:30.270269 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:30.270218 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:30.389572 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:30.389540 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:30.923776 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:30.923745 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx"] Jul 10 14:35:30.924010 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:30.923982 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" podUID="4ebf6590-457f-4b2f-b253-e805f17c409f" containerName="istio-proxy" containerID="cri-o://706357b6b8aa0ea0984816800c88a94aec18cf4fe4eb45557e22286eb2c7a693" gracePeriod=30 Jul 10 14:35:30.953149 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:30.953118 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:31.280260 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:31.280170 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:32.014729 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:32.014697 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:32.112762 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:32.112725 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:35.515001 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:35.514965 2576 patch_prober.go:28] interesting pod/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 14:35:35.515473 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:35.515041 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" podUID="4ebf6590-457f-4b2f-b253-e805f17c409f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 14:35:36.187083 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.187052 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:35:36.219922 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.219823 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-envoy\") pod \"4ebf6590-457f-4b2f-b253-e805f17c409f\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " Jul 10 14:35:36.219922 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.219883 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-credential-socket\") pod \"4ebf6590-457f-4b2f-b253-e805f17c409f\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " Jul 10 14:35:36.219922 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.219915 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-podinfo\") pod \"4ebf6590-457f-4b2f-b253-e805f17c409f\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " Jul 10 14:35:36.220200 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.219945 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-workload-certs\") pod \"4ebf6590-457f-4b2f-b253-e805f17c409f\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " Jul 10 14:35:36.220200 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.219961 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-data\") pod \"4ebf6590-457f-4b2f-b253-e805f17c409f\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " Jul 10 14:35:36.220200 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.219996 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebf6590-457f-4b2f-b253-e805f17c409f-istiod-ca-cert\") pod \"4ebf6590-457f-4b2f-b253-e805f17c409f\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " Jul 10 14:35:36.220200 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.220033 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m746w\" (UniqueName: \"kubernetes.io/projected/4ebf6590-457f-4b2f-b253-e805f17c409f-kube-api-access-m746w\") pod \"4ebf6590-457f-4b2f-b253-e805f17c409f\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " Jul 10 14:35:36.220200 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.220097 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-workload-socket\") pod \"4ebf6590-457f-4b2f-b253-e805f17c409f\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " Jul 10 14:35:36.220200 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.220141 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-token\") pod \"4ebf6590-457f-4b2f-b253-e805f17c409f\" (UID: \"4ebf6590-457f-4b2f-b253-e805f17c409f\") " Jul 10 14:35:36.220521 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.220298 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4ebf6590-457f-4b2f-b253-e805f17c409f" (UID: "4ebf6590-457f-4b2f-b253-e805f17c409f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:35:36.220521 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.220434 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-workload-certs\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:35:36.220521 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.220310 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4ebf6590-457f-4b2f-b253-e805f17c409f" (UID: "4ebf6590-457f-4b2f-b253-e805f17c409f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:35:36.220693 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.220523 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4ebf6590-457f-4b2f-b253-e805f17c409f" (UID: "4ebf6590-457f-4b2f-b253-e805f17c409f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:35:36.220931 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.220783 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebf6590-457f-4b2f-b253-e805f17c409f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4ebf6590-457f-4b2f-b253-e805f17c409f" (UID: "4ebf6590-457f-4b2f-b253-e805f17c409f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:35:36.221462 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.221434 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-data" (OuterVolumeSpecName: "istio-data") pod "4ebf6590-457f-4b2f-b253-e805f17c409f" (UID: "4ebf6590-457f-4b2f-b253-e805f17c409f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:35:36.222532 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.222489 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4ebf6590-457f-4b2f-b253-e805f17c409f" (UID: "4ebf6590-457f-4b2f-b253-e805f17c409f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:35:36.222644 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.222584 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4ebf6590-457f-4b2f-b253-e805f17c409f" (UID: "4ebf6590-457f-4b2f-b253-e805f17c409f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 14:35:36.222644 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.222592 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-token" (OuterVolumeSpecName: "istio-token") pod "4ebf6590-457f-4b2f-b253-e805f17c409f" (UID: "4ebf6590-457f-4b2f-b253-e805f17c409f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:35:36.222879 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.222851 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebf6590-457f-4b2f-b253-e805f17c409f-kube-api-access-m746w" (OuterVolumeSpecName: "kube-api-access-m746w") pod "4ebf6590-457f-4b2f-b253-e805f17c409f" (UID: "4ebf6590-457f-4b2f-b253-e805f17c409f"). InnerVolumeSpecName "kube-api-access-m746w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:35:36.321473 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.321429 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-workload-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:35:36.321473 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.321465 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-token\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:35:36.321473 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.321474 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-envoy\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:35:36.321473 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.321482 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-credential-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:35:36.321750 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.321491 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-podinfo\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:35:36.321750 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.321499 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebf6590-457f-4b2f-b253-e805f17c409f-istio-data\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:35:36.321750 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.321506 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebf6590-457f-4b2f-b253-e805f17c409f-istiod-ca-cert\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:35:36.321750 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:36.321515 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m746w\" (UniqueName: \"kubernetes.io/projected/4ebf6590-457f-4b2f-b253-e805f17c409f-kube-api-access-m746w\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:35:37.022584 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:37.022547 2576 generic.go:356] "Generic (PLEG): container finished" podID="4ebf6590-457f-4b2f-b253-e805f17c409f" containerID="706357b6b8aa0ea0984816800c88a94aec18cf4fe4eb45557e22286eb2c7a693" exitCode=0 Jul 10 14:35:37.023091 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:37.022600 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" Jul 10 14:35:37.023091 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:37.022621 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" event={"ID":"4ebf6590-457f-4b2f-b253-e805f17c409f","Type":"ContainerDied","Data":"706357b6b8aa0ea0984816800c88a94aec18cf4fe4eb45557e22286eb2c7a693"} Jul 10 14:35:37.023091 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:37.022679 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx" event={"ID":"4ebf6590-457f-4b2f-b253-e805f17c409f","Type":"ContainerDied","Data":"0b929df7e96e7a60a7f3f2b9433efe62582593db798afed7b221884ff9fc2ec1"} Jul 10 14:35:37.023091 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:37.022712 2576 scope.go:117] "RemoveContainer" containerID="706357b6b8aa0ea0984816800c88a94aec18cf4fe4eb45557e22286eb2c7a693" Jul 10 14:35:37.032189 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:37.032167 2576 scope.go:117] "RemoveContainer" containerID="ae2db71d417029ded22b2f71dae7a987ef5d54da54da085384cfdc74dfb6b21c" Jul 10 14:35:37.040573 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:37.040554 2576 scope.go:117] "RemoveContainer" containerID="706357b6b8aa0ea0984816800c88a94aec18cf4fe4eb45557e22286eb2c7a693" Jul 10 14:35:37.040891 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:35:37.040870 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"706357b6b8aa0ea0984816800c88a94aec18cf4fe4eb45557e22286eb2c7a693\": container with ID starting with 706357b6b8aa0ea0984816800c88a94aec18cf4fe4eb45557e22286eb2c7a693 not found: ID does not exist" containerID="706357b6b8aa0ea0984816800c88a94aec18cf4fe4eb45557e22286eb2c7a693" Jul 10 14:35:37.040963 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:37.040900 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706357b6b8aa0ea0984816800c88a94aec18cf4fe4eb45557e22286eb2c7a693"} err="failed to get container status \"706357b6b8aa0ea0984816800c88a94aec18cf4fe4eb45557e22286eb2c7a693\": rpc error: code = NotFound desc = could not find container \"706357b6b8aa0ea0984816800c88a94aec18cf4fe4eb45557e22286eb2c7a693\": container with ID starting with 706357b6b8aa0ea0984816800c88a94aec18cf4fe4eb45557e22286eb2c7a693 not found: ID does not exist" Jul 10 14:35:37.040963 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:37.040918 2576 scope.go:117] "RemoveContainer" containerID="ae2db71d417029ded22b2f71dae7a987ef5d54da54da085384cfdc74dfb6b21c" Jul 10 14:35:37.041103 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:35:37.041087 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2db71d417029ded22b2f71dae7a987ef5d54da54da085384cfdc74dfb6b21c\": container with ID starting with ae2db71d417029ded22b2f71dae7a987ef5d54da54da085384cfdc74dfb6b21c not found: ID does not exist" containerID="ae2db71d417029ded22b2f71dae7a987ef5d54da54da085384cfdc74dfb6b21c" Jul 10 14:35:37.041146 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:37.041107 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2db71d417029ded22b2f71dae7a987ef5d54da54da085384cfdc74dfb6b21c"} err="failed to get container status \"ae2db71d417029ded22b2f71dae7a987ef5d54da54da085384cfdc74dfb6b21c\": rpc error: code = NotFound desc = could not find container \"ae2db71d417029ded22b2f71dae7a987ef5d54da54da085384cfdc74dfb6b21c\": container with ID starting with ae2db71d417029ded22b2f71dae7a987ef5d54da54da085384cfdc74dfb6b21c not found: ID does not exist" Jul 10 14:35:37.045326 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:37.045300 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx"] Jul 10 14:35:37.051296 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:37.051276 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-e5410e-openshift-default-5db7f7d447-htggx"] Jul 10 14:35:39.035974 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:39.035944 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ebf6590-457f-4b2f-b253-e805f17c409f" path="/var/lib/kubelet/pods/4ebf6590-457f-4b2f-b253-e805f17c409f/volumes" Jul 10 14:35:41.961169 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:41.961125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:45.137455 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:45.137421 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hc6r7"] Jul 10 14:35:46.455587 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:46.455553 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:53.321733 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:53.321697 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:53.403731 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:53.403674 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:57.177283 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:57.177240 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw"] Jul 10 14:35:57.177898 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:57.177491 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" podUID="35d80e01-aa88-4182-bdbd-e1092352bc83" containerName="istio-proxy" containerID="cri-o://363a4fea357671e16d4efa301dff561242094064d4950b34a6a40c6373b24b39" gracePeriod=30 Jul 10 14:35:57.206112 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:57.206074 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:57.426911 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:57.426876 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:35:57.844420 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:35:57.844390 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c4f766d87-5vjgs_47da23a7-449a-4ce6-a77d-2c77eaaa3060/authorino/0.log" Jul 10 14:36:02.429053 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.429023 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:36:02.562729 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.562648 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2tkq\" (UniqueName: \"kubernetes.io/projected/35d80e01-aa88-4182-bdbd-e1092352bc83-kube-api-access-v2tkq\") pod \"35d80e01-aa88-4182-bdbd-e1092352bc83\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " Jul 10 14:36:02.562729 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.562693 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-envoy\") pod \"35d80e01-aa88-4182-bdbd-e1092352bc83\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " Jul 10 14:36:02.562729 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.562723 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-credential-socket\") pod \"35d80e01-aa88-4182-bdbd-e1092352bc83\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " Jul 10 14:36:02.563060 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.562741 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d80e01-aa88-4182-bdbd-e1092352bc83-istiod-ca-cert\") pod \"35d80e01-aa88-4182-bdbd-e1092352bc83\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " Jul 10 14:36:02.563060 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.562761 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-podinfo\") pod \"35d80e01-aa88-4182-bdbd-e1092352bc83\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " Jul 10 14:36:02.563060 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.562780 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-token\") pod \"35d80e01-aa88-4182-bdbd-e1092352bc83\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " Jul 10 14:36:02.563060 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.562798 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-data\") pod \"35d80e01-aa88-4182-bdbd-e1092352bc83\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " Jul 10 14:36:02.563060 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.562890 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-workload-socket\") pod \"35d80e01-aa88-4182-bdbd-e1092352bc83\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " Jul 10 14:36:02.563060 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.562935 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-workload-certs\") pod \"35d80e01-aa88-4182-bdbd-e1092352bc83\" (UID: \"35d80e01-aa88-4182-bdbd-e1092352bc83\") " Jul 10 14:36:02.563365 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.563124 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "35d80e01-aa88-4182-bdbd-e1092352bc83" (UID: "35d80e01-aa88-4182-bdbd-e1092352bc83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:36:02.563365 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.563228 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-credential-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:36:02.563365 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.563304 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35d80e01-aa88-4182-bdbd-e1092352bc83-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "35d80e01-aa88-4182-bdbd-e1092352bc83" (UID: "35d80e01-aa88-4182-bdbd-e1092352bc83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:36:02.563365 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.563346 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "35d80e01-aa88-4182-bdbd-e1092352bc83" (UID: "35d80e01-aa88-4182-bdbd-e1092352bc83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:36:02.563535 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.563443 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "35d80e01-aa88-4182-bdbd-e1092352bc83" (UID: "35d80e01-aa88-4182-bdbd-e1092352bc83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:36:02.563890 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.563843 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-data" (OuterVolumeSpecName: "istio-data") pod "35d80e01-aa88-4182-bdbd-e1092352bc83" (UID: "35d80e01-aa88-4182-bdbd-e1092352bc83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:36:02.565078 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.565045 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "35d80e01-aa88-4182-bdbd-e1092352bc83" (UID: "35d80e01-aa88-4182-bdbd-e1092352bc83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:36:02.565078 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.565066 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "35d80e01-aa88-4182-bdbd-e1092352bc83" (UID: "35d80e01-aa88-4182-bdbd-e1092352bc83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 14:36:02.565238 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.565116 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d80e01-aa88-4182-bdbd-e1092352bc83-kube-api-access-v2tkq" (OuterVolumeSpecName: "kube-api-access-v2tkq") pod "35d80e01-aa88-4182-bdbd-e1092352bc83" (UID: "35d80e01-aa88-4182-bdbd-e1092352bc83"). InnerVolumeSpecName "kube-api-access-v2tkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:36:02.565326 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.565308 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-token" (OuterVolumeSpecName: "istio-token") pod "35d80e01-aa88-4182-bdbd-e1092352bc83" (UID: "35d80e01-aa88-4182-bdbd-e1092352bc83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:36:02.664464 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.664414 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2tkq\" (UniqueName: \"kubernetes.io/projected/35d80e01-aa88-4182-bdbd-e1092352bc83-kube-api-access-v2tkq\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:36:02.664464 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.664457 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-envoy\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:36:02.664464 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.664467 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d80e01-aa88-4182-bdbd-e1092352bc83-istiod-ca-cert\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:36:02.664464 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.664476 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-podinfo\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:36:02.664464 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.664484 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-token\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:36:02.664888 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.664495 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-istio-data\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:36:02.664888 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.664503 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-workload-socket\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:36:02.664888 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:02.664513 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d80e01-aa88-4182-bdbd-e1092352bc83-workload-certs\") on node \"ip-10-0-143-25.ec2.internal\" DevicePath \"\"" Jul 10 14:36:03.089231 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:03.089194 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6j79w_4f1fd144-6cfd-458c-af0a-0139608099a0/server/0.log" Jul 10 14:36:03.155538 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:03.155504 2576 generic.go:356] "Generic (PLEG): container finished" podID="35d80e01-aa88-4182-bdbd-e1092352bc83" containerID="363a4fea357671e16d4efa301dff561242094064d4950b34a6a40c6373b24b39" exitCode=0 Jul 10 14:36:03.155716 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:03.155563 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" Jul 10 14:36:03.155716 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:03.155587 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" event={"ID":"35d80e01-aa88-4182-bdbd-e1092352bc83","Type":"ContainerDied","Data":"363a4fea357671e16d4efa301dff561242094064d4950b34a6a40c6373b24b39"} Jul 10 14:36:03.155716 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:03.155652 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw" event={"ID":"35d80e01-aa88-4182-bdbd-e1092352bc83","Type":"ContainerDied","Data":"788cc84ae724e107d16484937cc9a8703cf24911ab8908fabab8069fe9c196e5"} Jul 10 14:36:03.155716 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:03.155682 2576 scope.go:117] "RemoveContainer" containerID="363a4fea357671e16d4efa301dff561242094064d4950b34a6a40c6373b24b39" Jul 10 14:36:03.164668 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:03.164650 2576 scope.go:117] "RemoveContainer" containerID="9e0b60c0cee1be2e5a3f57eec0f293098ea53ceb3f192213fedf5e40a911faf0" Jul 10 14:36:03.175009 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:03.174988 2576 scope.go:117] "RemoveContainer" containerID="363a4fea357671e16d4efa301dff561242094064d4950b34a6a40c6373b24b39" Jul 10 14:36:03.175628 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:36:03.175602 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"363a4fea357671e16d4efa301dff561242094064d4950b34a6a40c6373b24b39\": container with ID starting with 363a4fea357671e16d4efa301dff561242094064d4950b34a6a40c6373b24b39 not found: ID does not exist" containerID="363a4fea357671e16d4efa301dff561242094064d4950b34a6a40c6373b24b39" Jul 10 14:36:03.175748 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:03.175635 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363a4fea357671e16d4efa301dff561242094064d4950b34a6a40c6373b24b39"} err="failed to get container status \"363a4fea357671e16d4efa301dff561242094064d4950b34a6a40c6373b24b39\": rpc error: code = NotFound desc = could not find container \"363a4fea357671e16d4efa301dff561242094064d4950b34a6a40c6373b24b39\": container with ID starting with 363a4fea357671e16d4efa301dff561242094064d4950b34a6a40c6373b24b39 not found: ID does not exist" Jul 10 14:36:03.175748 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:03.175655 2576 scope.go:117] "RemoveContainer" containerID="9e0b60c0cee1be2e5a3f57eec0f293098ea53ceb3f192213fedf5e40a911faf0" Jul 10 14:36:03.175983 ip-10-0-143-25 kubenswrapper[2576]: E0710 14:36:03.175962 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0b60c0cee1be2e5a3f57eec0f293098ea53ceb3f192213fedf5e40a911faf0\": container with ID starting with 9e0b60c0cee1be2e5a3f57eec0f293098ea53ceb3f192213fedf5e40a911faf0 not found: ID does not exist" containerID="9e0b60c0cee1be2e5a3f57eec0f293098ea53ceb3f192213fedf5e40a911faf0" Jul 10 14:36:03.176028 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:03.175989 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0b60c0cee1be2e5a3f57eec0f293098ea53ceb3f192213fedf5e40a911faf0"} err="failed to get container status \"9e0b60c0cee1be2e5a3f57eec0f293098ea53ceb3f192213fedf5e40a911faf0\": rpc error: code = NotFound desc = could not find container \"9e0b60c0cee1be2e5a3f57eec0f293098ea53ceb3f192213fedf5e40a911faf0\": container with ID starting with 9e0b60c0cee1be2e5a3f57eec0f293098ea53ceb3f192213fedf5e40a911faf0 not found: ID does not exist" Jul 10 14:36:03.178455 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:03.178432 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw"] Jul 10 14:36:03.180226 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:03.180206 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-47e604-openshift-default-86d6c5d995-w2msw"] Jul 10 14:36:03.340522 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:03.340448 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-545b4db9db-g84xk_21d44d5e-e3bd-426a-8fa6-e40557b41a9c/copy-manifests/0.log" Jul 10 14:36:03.354059 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:03.354025 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-545b4db9db-g84xk_21d44d5e-e3bd-426a-8fa6-e40557b41a9c/manager/0.log" Jul 10 14:36:04.074549 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:04.074523 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29728215-jfbnn_52beecd1-c181-46b5-be0f-07b31ea49dd8/cleanup/0.log" Jul 10 14:36:04.286289 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:04.286260 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-561ce10f-bgm49_49293a72-c240-4d30-9978-82945f1dedc6/revoke-keys/0.log" Jul 10 14:36:04.396552 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:04.396521 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-b78b9ecf-qggp9_bdd3c27f-4e0b-4a5a-b140-f6b27d5ada36/revoke-keys/0.log" Jul 10 14:36:04.736101 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:04.736012 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-cce0783f-sbxq5_54e7a6c2-f4e0-424a-91c8-b702e31983f1/revoke-keys/0.log" Jul 10 14:36:04.839761 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:04.839735 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-ee9f0f-zqnpm_00839295-b0d3-414b-a161-9d210f3e8917/revoke-keys/0.log" Jul 10 14:36:05.036398 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:05.036313 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35d80e01-aa88-4182-bdbd-e1092352bc83" path="/var/lib/kubelet/pods/35d80e01-aa88-4182-bdbd-e1092352bc83/volumes" Jul 10 14:36:05.063459 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:05.063427 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-0c74fa54-j87j4_9a58b6a9-fff8-4a71-9ac4-9ed6be2bc395/revoke-keys/0.log" Jul 10 14:36:05.186206 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:05.186177 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-a6e9e1cb-hc6l2_a484a423-ef61-4798-bf44-a0cb98767f14/revoke-keys/0.log" Jul 10 14:36:06.805474 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:06.805444 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c4f766d87-5vjgs_47da23a7-449a-4ce6-a77d-2c77eaaa3060/authorino/0.log" Jul 10 14:36:07.150826 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:07.150788 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-swbkn_a1078386-e19c-4f8e-bf73-36ba6db1cadb/kuadrant-console-plugin/0.log" Jul 10 14:36:07.553856 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:07.553755 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-hc6r7_0e9d79f9-68a9-484a-bbf2-1ca5b5228d09/limitador/0.log" Jul 10 14:36:08.042365 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:08.042334 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4jvlz6_afc01637-a228-4494-8ef8-cc79be42da25/istio-proxy/0.log" Jul 10 14:36:08.489021 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:08.488980 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8tjln_6ad868c2-3722-4d72-883c-1e1780c6ddce/istio-proxy/1.log" Jul 10 14:36:09.311240 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:09.311213 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-r7vzk_50bdcb80-0e45-47dc-b78f-846cbf652c4b/main/0.log" Jul 10 14:36:09.669372 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:09.669345 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-5h9jw_31377e37-f8eb-4e20-87ef-22ac212289a0/main/0.log" Jul 10 14:36:14.252191 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.252157 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2vrfh/must-gather-zggcv"] Jul 10 14:36:14.252653 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.252470 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ebf6590-457f-4b2f-b253-e805f17c409f" containerName="istio-proxy" Jul 10 14:36:14.252653 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.252480 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ebf6590-457f-4b2f-b253-e805f17c409f" containerName="istio-proxy" Jul 10 14:36:14.252653 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.252502 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35d80e01-aa88-4182-bdbd-e1092352bc83" containerName="istio-proxy" Jul 10 14:36:14.252653 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.252507 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="35d80e01-aa88-4182-bdbd-e1092352bc83" containerName="istio-proxy" Jul 10 14:36:14.252653 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.252516 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ebf6590-457f-4b2f-b253-e805f17c409f" containerName="istio-proxy" Jul 10 14:36:14.252653 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.252523 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ebf6590-457f-4b2f-b253-e805f17c409f" containerName="istio-proxy" Jul 10 14:36:14.252653 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.252533 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2fdf737-7866-4da4-82ee-cb521b2d2ceb" containerName="test-curl-7289-140576863133536" Jul 10 14:36:14.252653 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.252540 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2fdf737-7866-4da4-82ee-cb521b2d2ceb" containerName="test-curl-7289-140576863133536" Jul 10 14:36:14.252653 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.252547 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35d80e01-aa88-4182-bdbd-e1092352bc83" containerName="istio-proxy" Jul 10 14:36:14.252653 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.252552 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="35d80e01-aa88-4182-bdbd-e1092352bc83" containerName="istio-proxy" Jul 10 14:36:14.252653 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.252601 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="35d80e01-aa88-4182-bdbd-e1092352bc83" containerName="istio-proxy" Jul 10 14:36:14.252653 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.252608 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ebf6590-457f-4b2f-b253-e805f17c409f" containerName="istio-proxy" Jul 10 14:36:14.252653 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.252615 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2fdf737-7866-4da4-82ee-cb521b2d2ceb" containerName="test-curl-7289-140576863133536" Jul 10 14:36:14.252653 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.252620 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="35d80e01-aa88-4182-bdbd-e1092352bc83" containerName="istio-proxy" Jul 10 14:36:14.253166 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.252781 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ebf6590-457f-4b2f-b253-e805f17c409f" containerName="istio-proxy" Jul 10 14:36:14.256942 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.256925 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vrfh/must-gather-zggcv" Jul 10 14:36:14.259542 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.259518 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vrfh\"/\"openshift-service-ca.crt\"" Jul 10 14:36:14.259664 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.259633 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vrfh\"/\"kube-root-ca.crt\"" Jul 10 14:36:14.260149 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.260135 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2vrfh\"/\"default-dockercfg-cn9ln\"" Jul 10 14:36:14.263380 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.263308 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc7sd\" (UniqueName: \"kubernetes.io/projected/a8a0dee6-71af-481a-b660-af99cf8ebe68-kube-api-access-wc7sd\") pod \"must-gather-zggcv\" (UID: \"a8a0dee6-71af-481a-b660-af99cf8ebe68\") " pod="openshift-must-gather-2vrfh/must-gather-zggcv" Jul 10 14:36:14.263380 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.263353 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a8a0dee6-71af-481a-b660-af99cf8ebe68-must-gather-output\") pod \"must-gather-zggcv\" (UID: \"a8a0dee6-71af-481a-b660-af99cf8ebe68\") " pod="openshift-must-gather-2vrfh/must-gather-zggcv" Jul 10 14:36:14.270135 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.270108 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vrfh/must-gather-zggcv"] Jul 10 14:36:14.275067 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.275043 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:36:14.364012 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.363964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc7sd\" (UniqueName: \"kubernetes.io/projected/a8a0dee6-71af-481a-b660-af99cf8ebe68-kube-api-access-wc7sd\") pod \"must-gather-zggcv\" (UID: \"a8a0dee6-71af-481a-b660-af99cf8ebe68\") " pod="openshift-must-gather-2vrfh/must-gather-zggcv" Jul 10 14:36:14.364205 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.364040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a8a0dee6-71af-481a-b660-af99cf8ebe68-must-gather-output\") pod \"must-gather-zggcv\" (UID: \"a8a0dee6-71af-481a-b660-af99cf8ebe68\") " pod="openshift-must-gather-2vrfh/must-gather-zggcv" Jul 10 14:36:14.364403 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.364384 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a8a0dee6-71af-481a-b660-af99cf8ebe68-must-gather-output\") pod \"must-gather-zggcv\" (UID: \"a8a0dee6-71af-481a-b660-af99cf8ebe68\") " pod="openshift-must-gather-2vrfh/must-gather-zggcv" Jul 10 14:36:14.375366 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.375341 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc7sd\" (UniqueName: \"kubernetes.io/projected/a8a0dee6-71af-481a-b660-af99cf8ebe68-kube-api-access-wc7sd\") pod \"must-gather-zggcv\" (UID: \"a8a0dee6-71af-481a-b660-af99cf8ebe68\") " pod="openshift-must-gather-2vrfh/must-gather-zggcv" Jul 10 14:36:14.378403 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.378368 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:36:14.566617 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.566525 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vrfh/must-gather-zggcv" Jul 10 14:36:14.720456 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:14.720430 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vrfh/must-gather-zggcv"] Jul 10 14:36:14.723385 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:36:14.723353 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8a0dee6_71af_481a_b660_af99cf8ebe68.slice/crio-a92b9949898f158e361558541ef46ab6a4f7023101a357c2e1afe2dd2d627ed2 WatchSource:0}: Error finding container a92b9949898f158e361558541ef46ab6a4f7023101a357c2e1afe2dd2d627ed2: Status 404 returned error can't find the container with id a92b9949898f158e361558541ef46ab6a4f7023101a357c2e1afe2dd2d627ed2 Jul 10 14:36:15.216502 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:15.216465 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vrfh/must-gather-zggcv" event={"ID":"a8a0dee6-71af-481a-b660-af99cf8ebe68","Type":"ContainerStarted","Data":"a92b9949898f158e361558541ef46ab6a4f7023101a357c2e1afe2dd2d627ed2"} Jul 10 14:36:16.226713 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:16.226667 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vrfh/must-gather-zggcv" event={"ID":"a8a0dee6-71af-481a-b660-af99cf8ebe68","Type":"ContainerStarted","Data":"c6c30d0929543943b1079fbc631301d7d171a1c1d4133d899bdb2682c1768f02"} Jul 10 14:36:16.227151 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:16.226729 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vrfh/must-gather-zggcv" event={"ID":"a8a0dee6-71af-481a-b660-af99cf8ebe68","Type":"ContainerStarted","Data":"2aaf555b433adb37d8c3cdf7ddef1a4400f16defaa7a0d0e3156d22cf84a231a"} Jul 10 14:36:17.235594 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:17.235560 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zbx2d_c708f7f5-e4fa-4dd0-9fb3-aa4d669e98f4/global-pull-secret-syncer/0.log" Jul 10 14:36:17.297543 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:17.297506 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" Jul 10 14:36:17.450622 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:17.450592 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-25.ec2.internal_b15c5c2738510537812bfa649878a461/haproxy/0.log" Jul 10 14:36:22.723758 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:22.723389 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c4f766d87-5vjgs_47da23a7-449a-4ce6-a77d-2c77eaaa3060/authorino/0.log" Jul 10 14:36:22.813942 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:22.813907 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-swbkn_a1078386-e19c-4f8e-bf73-36ba6db1cadb/kuadrant-console-plugin/0.log" Jul 10 14:36:23.092146 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:23.091927 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-hc6r7_0e9d79f9-68a9-484a-bbf2-1ca5b5228d09/limitador/0.log" Jul 10 14:36:25.225465 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:25.225429 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-65k6j_6b79eb5f-3947-4b51-b06b-f96ba7ed8e07/cluster-monitoring-operator/0.log" Jul 10 14:36:25.496783 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:25.496710 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gqz9s_1e8b6f70-5e39-44e5-8475-803fe93dd246/node-exporter/0.log" Jul 10 14:36:25.526663 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:25.526635 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gqz9s_1e8b6f70-5e39-44e5-8475-803fe93dd246/kube-rbac-proxy/0.log" Jul 10 14:36:25.553504 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:25.553475 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gqz9s_1e8b6f70-5e39-44e5-8475-803fe93dd246/init-textfile/0.log" Jul 10 14:36:26.268565 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:26.268532 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:36:27.323382 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.323323 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2vrfh/must-gather-zggcv" podStartSLOduration=12.559152114 podStartE2EDuration="13.323304158s" totalImagesPullingTime="764.152044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:36:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:36:16.246004349 +0000 UTC m=+2783.709609797" watchObservedRunningTime="2026-07-10 14:36:27.323304158 +0000 UTC m=+2794.786909657" Jul 10 14:36:27.324399 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.324373 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt"] Jul 10 14:36:27.329868 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.329794 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:27.353838 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.353786 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt"] Jul 10 14:36:27.489106 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.489072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wgt4\" (UniqueName: \"kubernetes.io/projected/58ca808a-b1b1-424f-a45d-5abde5f50dae-kube-api-access-4wgt4\") pod \"perf-node-gather-daemonset-7tctt\" (UID: \"58ca808a-b1b1-424f-a45d-5abde5f50dae\") " pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:27.489106 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.489111 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/58ca808a-b1b1-424f-a45d-5abde5f50dae-proc\") pod \"perf-node-gather-daemonset-7tctt\" (UID: \"58ca808a-b1b1-424f-a45d-5abde5f50dae\") " pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:27.489325 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.489167 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58ca808a-b1b1-424f-a45d-5abde5f50dae-lib-modules\") pod \"perf-node-gather-daemonset-7tctt\" (UID: \"58ca808a-b1b1-424f-a45d-5abde5f50dae\") " pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:27.489325 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.489234 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/58ca808a-b1b1-424f-a45d-5abde5f50dae-podres\") pod \"perf-node-gather-daemonset-7tctt\" (UID: \"58ca808a-b1b1-424f-a45d-5abde5f50dae\") " pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:27.489325 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.489252 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58ca808a-b1b1-424f-a45d-5abde5f50dae-sys\") pod \"perf-node-gather-daemonset-7tctt\" (UID: \"58ca808a-b1b1-424f-a45d-5abde5f50dae\") " pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:27.589887 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.589774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/58ca808a-b1b1-424f-a45d-5abde5f50dae-podres\") pod \"perf-node-gather-daemonset-7tctt\" (UID: \"58ca808a-b1b1-424f-a45d-5abde5f50dae\") " pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:27.590103 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.590065 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/58ca808a-b1b1-424f-a45d-5abde5f50dae-podres\") pod \"perf-node-gather-daemonset-7tctt\" (UID: \"58ca808a-b1b1-424f-a45d-5abde5f50dae\") " pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:27.590248 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.590112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58ca808a-b1b1-424f-a45d-5abde5f50dae-sys\") pod \"perf-node-gather-daemonset-7tctt\" (UID: \"58ca808a-b1b1-424f-a45d-5abde5f50dae\") " pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:27.590248 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.590214 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58ca808a-b1b1-424f-a45d-5abde5f50dae-sys\") pod \"perf-node-gather-daemonset-7tctt\" (UID: \"58ca808a-b1b1-424f-a45d-5abde5f50dae\") " pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:27.590372 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.590262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wgt4\" (UniqueName: \"kubernetes.io/projected/58ca808a-b1b1-424f-a45d-5abde5f50dae-kube-api-access-4wgt4\") pod \"perf-node-gather-daemonset-7tctt\" (UID: \"58ca808a-b1b1-424f-a45d-5abde5f50dae\") " pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:27.590372 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.590293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/58ca808a-b1b1-424f-a45d-5abde5f50dae-proc\") pod \"perf-node-gather-daemonset-7tctt\" (UID: \"58ca808a-b1b1-424f-a45d-5abde5f50dae\") " pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:27.590372 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.590335 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58ca808a-b1b1-424f-a45d-5abde5f50dae-lib-modules\") pod \"perf-node-gather-daemonset-7tctt\" (UID: \"58ca808a-b1b1-424f-a45d-5abde5f50dae\") " pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:27.590533 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.590436 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/58ca808a-b1b1-424f-a45d-5abde5f50dae-proc\") pod \"perf-node-gather-daemonset-7tctt\" (UID: \"58ca808a-b1b1-424f-a45d-5abde5f50dae\") " pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:27.590533 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.590507 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58ca808a-b1b1-424f-a45d-5abde5f50dae-lib-modules\") pod \"perf-node-gather-daemonset-7tctt\" (UID: \"58ca808a-b1b1-424f-a45d-5abde5f50dae\") " pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:27.605515 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.603984 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wgt4\" (UniqueName: \"kubernetes.io/projected/58ca808a-b1b1-424f-a45d-5abde5f50dae-kube-api-access-4wgt4\") pod \"perf-node-gather-daemonset-7tctt\" (UID: \"58ca808a-b1b1-424f-a45d-5abde5f50dae\") " pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:27.643863 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.643824 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:27.835156 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:27.835100 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt"] Jul 10 14:36:27.839296 ip-10-0-143-25 kubenswrapper[2576]: W0710 14:36:27.839251 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod58ca808a_b1b1_424f_a45d_5abde5f50dae.slice/crio-3bf6e24d49279e041ed0b2b4af8adb7a64035dcc37297c4c3e5101a1eb06a2da WatchSource:0}: Error finding container 3bf6e24d49279e041ed0b2b4af8adb7a64035dcc37297c4c3e5101a1eb06a2da: Status 404 returned error can't find the container with id 3bf6e24d49279e041ed0b2b4af8adb7a64035dcc37297c4c3e5101a1eb06a2da Jul 10 14:36:28.268617 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:28.268589 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/2.log" Jul 10 14:36:28.273037 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:28.273001 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vpmbq_3b42fcdc-6276-479f-b2d6-35b4044878f6/console-operator/3.log" Jul 10 14:36:28.313368 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:28.313322 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" event={"ID":"58ca808a-b1b1-424f-a45d-5abde5f50dae","Type":"ContainerStarted","Data":"a66449743de360efd134e4f6d4b349fce54d1e7715bf96c6362174d8b480926c"} Jul 10 14:36:28.313562 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:28.313394 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" event={"ID":"58ca808a-b1b1-424f-a45d-5abde5f50dae","Type":"ContainerStarted","Data":"3bf6e24d49279e041ed0b2b4af8adb7a64035dcc37297c4c3e5101a1eb06a2da"} Jul 10 14:36:28.314267 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:28.314239 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:28.344767 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:28.344704 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" podStartSLOduration=1.344680425 podStartE2EDuration="1.344680425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 14:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:36:28.343475609 +0000 UTC m=+2795.807081266" watchObservedRunningTime="2026-07-10 14:36:28.344680425 +0000 UTC m=+2795.808285873" Jul 10 14:36:29.542970 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:29.542943 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-whjfm_cc5b6a43-58f9-4257-a621-613436e24b9d/volume-data-source-validator/0.log" Jul 10 14:36:30.860305 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:30.860276 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qczvj_1c402cd2-eb3e-4833-98d9-32cfda11a80b/dns/0.log" Jul 10 14:36:30.885471 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:30.885442 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qczvj_1c402cd2-eb3e-4833-98d9-32cfda11a80b/kube-rbac-proxy/0.log" Jul 10 14:36:30.976855 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:30.976822 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v6qp5_3c3e7158-27d5-4642-b99f-7f2a8a728726/dns-node-resolver/0.log" Jul 10 14:36:31.609493 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:31.609445 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9nsfh_868d4bcd-6eec-475b-a138-12fa9712ae87/node-ca/0.log" Jul 10 14:36:32.527355 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:32.527300 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wgd6_aea6d43a-5030-4d5b-89e7-c8dfd95b0b27/konnectivity-agent/0.log" limit=1024 Jul 10 14:36:32.729095 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:32.729058 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4jvlz6_afc01637-a228-4494-8ef8-cc79be42da25/istio-proxy/0.log" Jul 10 14:36:32.942594 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:32.942557 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8tjln_6ad868c2-3722-4d72-883c-1e1780c6ddce/istio-proxy/0.log" Jul 10 14:36:32.978747 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:32.978714 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8tjln_6ad868c2-3722-4d72-883c-1e1780c6ddce/istio-proxy/1.log" Jul 10 14:36:34.082453 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:34.082425 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l2hjm_9ce29e96-a26b-44e6-8c74-709f97aa2418/serve-healthcheck-canary/0.log" Jul 10 14:36:34.809045 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:34.809012 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n4ddz_5fef8787-631b-4b5e-9330-15d0f084479d/kube-rbac-proxy/0.log" Jul 10 14:36:34.833406 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:34.833374 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n4ddz_5fef8787-631b-4b5e-9330-15d0f084479d/exporter/0.log" Jul 10 14:36:34.856540 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:34.856502 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n4ddz_5fef8787-631b-4b5e-9330-15d0f084479d/extractor/0.log" Jul 10 14:36:35.342476 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:35.342441 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2vrfh/perf-node-gather-daemonset-7tctt" Jul 10 14:36:37.822203 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:37.822177 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6j79w_4f1fd144-6cfd-458c-af0a-0139608099a0/server/0.log" Jul 10 14:36:37.922538 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:37.922508 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4jvlz6_afc01637-a228-4494-8ef8-cc79be42da25/istio-proxy/0.log" Jul 10 14:36:37.995971 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:37.995937 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-545b4db9db-g84xk_21d44d5e-e3bd-426a-8fa6-e40557b41a9c/manager/0.log" Jul 10 14:36:38.021749 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:38.021723 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-545b4db9db-g84xk_21d44d5e-e3bd-426a-8fa6-e40557b41a9c/copy-manifests/0.log" Jul 10 14:36:38.091497 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:38.091421 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8tjln_6ad868c2-3722-4d72-883c-1e1780c6ddce/istio-proxy/0.log" Jul 10 14:36:38.122038 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:38.122001 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8tjln_6ad868c2-3722-4d72-883c-1e1780c6ddce/istio-proxy/1.log" Jul 10 14:36:39.523281 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:39.523247 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-745c8d65dc-z65kk_557d8d62-4252-4155-af82-2c3dbe914f21/manager/0.log" Jul 10 14:36:39.556363 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:39.556280 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-cqp6x_92467d01-6dac-4288-9232-70b7456ab773/openshift-lws-operator/0.log" Jul 10 14:36:39.626362 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:39.626329 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4jvlz6_afc01637-a228-4494-8ef8-cc79be42da25/istio-proxy/0.log" Jul 10 14:36:39.805391 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:39.805364 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8tjln_6ad868c2-3722-4d72-883c-1e1780c6ddce/istio-proxy/0.log" Jul 10 14:36:39.845384 ip-10-0-143-25 kubenswrapper[2576]: I0710 14:36:39.845302 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8tjln_6ad868c2-3722-4d72-883c-1e1780c6ddce/istio-proxy/1.log"